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F-35  SUSTAINMENT 

DOD  Needs  a  Plan  to  Address  Risks  Related  to  Its 
Central  Logistics  System 


Why  GAO  Did  This  Study 

The  F-35  is  the  most  ambitious  and 
expensive  weapon  system  in  DOD’s 
history,  with  sustainment  costs 
comprising  the  vast  majority  of  DOD’s 
$1.3  trillion  cost  estimate.  Central  to 
F-35  sustainment  is  ALIS — a  complex 
system  supporting  operations,  mission 
planning,  supply-chain  management, 
maintenance,  and  other  processes. 
The  F-35  program  is  approaching 
several  key  milestones:  the  Air  Force 
and  Navy  are  to  declare  the  ability  to 
operate  and  deploy  the  F-35  in  2016 
and  2018  respectively,  and  full-rate 
production  of  the  aircraft  is  to  begin  in 
2019.  Flowever,  ALIS  has  experienced 
developmental  issues  and  schedule 
delays  that  have  put  aircraft  availability 
and  flying  missions  at  risk.  The 
National  Defense  Authorization  Act  for 
Fiscal  Year  2016  included  a  provision 
that  GAO  review  the  F-35’s  ALIS.  This 
report  assesses,  among  other  things, 
the  extent  to  which  DOD  has  (1)  a  plan 
to  ensure  that  ALIS  is  fully  functional 
as  key  program  milestones  approach 
and  (2)  credibly  and  accurately 
estimated  ALIS  costs.  GAO  reviewed 
F-35  program  documentation, 
interviewed  officials,  and  conducted 
focus  groups  with  ALIS  users. 

What  GAO  Recommends 

GAO  is  making  four  recommendations 
including  that  DOD  develop  a  plan  to 
address  ALIS  risks,  and  conduct 
certain  analyses  and  include  historical 
data  to  improve  its  ALIS  cost  estimate. 
DOD  concurred  with  developing  a  plan 
and  partially  concurred  with  the  cost 
estimating  recommendations,  stating 
that  it  follows  its  own  guidance.  GAO 
continues  to  believe  the 
recommendations  are  valid,  as 
discussed  in  the  report. 

View  GAO-16-439.  For  more  information, 
contact  Cary  Russell  at  (202)  512-5431  or 
russellc@gao.gov 


What  GAO  Found 

The  Department  of  Defense  (DOD)  is  aware  of  risks  that  could  affect  the  F-35’s 
Autonomic  Logistics  Information  System  (ALIS),  but  does  not  have  a  plan  to 
ensure  that  ALIS  is  fully  functional  as  key  program  milestones  approach.  ALIS 
users,  including  pilots  and  maintainers,  in  GAO’s  focus  groups  identified  benefits 
of  the  system,  such  as  the  incorporation  of  multiple  functions  into  a  single 
system.  However,  users  also  identified  several  issues  that  could  result  in 
operational  and  schedule  risks.  These  include  the  following: 

•  ALIS  may  not  be  deployable:  ALIS  requires  server  connectivity  and  the 
necessary  infrastructure  to  provide  power  to  the  system.  The  Marine  Corps, 
which  often  deploys  to  austere  locations,  declared  in  July  2015  its  ability  to 
operate  and  deploy  the  F-35  without  conducting  deployability  tests  of  ALIS.  A 
newer  version  of  ALIS  was  put  into  operation  in  the  summer  of  2015,  but 
DOD  has  not  yet  completed  comprehensive  deployability  tests. 

•  ALIS  does  not  have  redundant  infrastructure:  ALIS’s  current  design 
results  in  all  F-35  data  produced  across  the  U.S.  fleet  to  be  routed  to  a 
Central  Point  of  Entry  and  then  to  ALIS’s  main  operating  unit  with  no  backup 
system  or  redundancy.  If  either  of  these  fail,  it  could  take  the  entire  F-35  fleet 
offline. 

DOD  is  taking  some  steps  to  address  these  and  other  risks  such  as  resolving 
smaller  ALIS  functionality  issues  between  major  software  upgrades  and 
considering  the  procurement  of  additional  ALIS  infrastructure  but  the  department 
is  attending  to  issues  on  a  case-by-case  basis.  DOD  does  not  have  a  plan  that 
prioritizes  ALIS  risks  to  ensure  that  the  most  important  are  expediently 
addressed  and  that  DOD  has  a  fully  functional  ALIS  as  program  milestones  draw 
close.  By  continuing  to  respond  to  issues  on  a  case-by-case  basis  rather  than  in 
a  holistic  manner,  there  is  no  guarantee  that  DOD  will  address  the  highest  risks 
by  the  start  of  full-rate  production  in  2019,  and  as  a  result,  DOD  may  encounter 
further  schedule  and  development  delays,  which  could  affect  operations  and 
potentially  lead  to  cost  increases. 

DOD  has  estimated  total  ALIS  costs  to  be  about  $16.7  billion  over  the  F-35’s 
56-year  life  cycle,  but  performing  additional  analyses  and  including  historical  cost 
data  would  increase  the  credibility  and  accuracy  of  DOD’s  estimate.  GAO’s  cost 
estimating  best  practices  state  that  cost  estimates  should  include  uncertainty 
analyses  to  determine  the  level  of  uncertainty  associated  with  the  estimate  in 
order  to  be  credible.  In  addition,  credible  cost  estimates  should  include  sensitivity 
analyses  to  examine  how  changes  to  individual  assumptions  and  inputs  affect 
the  estimate  as  a  whole.  DOD's  guidance  does  not  require  the  department  to 
perform  these  analyses  for  ALIS,  and  DOD  officials  stated  that  they  have  not 
done  so  in  part  because  ALIS  constitutes  less  than  2  percent  of  the  F-35’s 
estimated  total  sustainment  costs.  Program  officials  said  that  if  ALIS  is  not  fully 
functional,  the  F-35  could  not  be  operated  as  frequently  as  intended,  but  a  DOD- 
commissioned  plan  found  that  schedule  slippage  and  functionality  problems  with 
ALIS  could  lead  to  $20-100  billion  in  additional  costs.  Without  uncertainty  and 
sensitivity  analyses,  it  is  unclear  how  ALIS  can  affect  costs.  GAO  also  found  that 
using  historical  cost  data  would  make  DOD’s  cost  estimate  more  accurate. 
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AR 
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Office  of  the  Secretary  of  Defense 
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TMS 
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April  14,  2016 
Congressional  Committees 

The  Department  of  Defense  (DOD)  calls  the  F-35  Lightning  II  the 
backbone  of  U.S.  air  combat  superiority  for  decades  to  come,  as  it  plans 
for  the  aircraft  to  replace  the  legacy  tactical  fighter  fleets  of  the  Air  Force, 
Navy,  and  Marine  Corps.  With  estimated  total  ownership  costs  of  just 
under  $1.3  trillion  over  its  56-year  life  cycle,  the  F-35  is  the  most 
ambitious  and  costly  program  in  DOD’s  history.1  The  cost  to  sustain  the 
aircraft — estimated  to  be  just  under  $900  billion — is  the  primary  cost 
driver  for  the  F-35  program.  Central  to  F-35  sustainment  is  the  Autonomic 
Logistics  Information  System  (ALIS) — a  complex  system  that  supports 
operations,  mission  planning,  supply-chain  management,  maintenance, 
and  other  processes.  A  DOD  official  has  described  ALIS  as  the  brains  of 
the  aircraft,  calling  it  one  of  the  three  major  components  that  make  up  the 
F-35,  along  with  the  airframe  and  engine.  However,  ALIS  has 
experienced  developmental  issues,  including  system  functionality 
problems,  and  schedule  delays  that  have  put  key  performance 
parameters,  such  as  aircraft  availability  and  flying  missions,  at  risk.  The 
F-35  program  is  approaching  several  key  milestones:  the  Air  Force  and 
Navy  are  to  declare  the  ability  to  operate  and  deploy  the  aircraft  (“initial 
operational  capability”)  in  2016  and  2018  respectively,  and  full-rate 
production  decision  of  the  program  is  planned  for  201 9. 2  Recognizing  that 
a  fully  functional  ALIS  is  critical  to  the  program’s  overall  success,  in 
October  2015,  the  F-35  executive  program  officer  testified  before 
Congress  that  ALIS  is  one  of  the  most  significant  technical  and  schedule 
risks  to  the  program.3 


^he  $1 .3  trillion  dollar  total  ownership  cost  and  all  other  cost  estimates  throughout  this 
report  are  based  on  the  2014  F-35  program  office  cost  estimate. 

2The  Marine  Corps  declared  initial  operational  capability  in  July  2015.  The  full-rate 
production  is  a  decision  following  the  completion  of  operational  testing,  to  scale  up 
production  and  fielding.  During  full-rate  production,  the  remaining  production  or 
deployment  of  the  product  is  completed,  leading  to  full  operational  capability  or  full 
deployment. 

3F-35  Lightning  II  Program  Update,  testimony  before  the  Subcommittee  on  Tactical  Air 
and  Land  Forces,  House  Committee  on  Armed  Services  H.  Comm,  on  Armed  Services, 
Subcomm.  on  Tactical  Air  and  Land  Forces,  114th  Cong.,  1st  session  (October21, 2015). 
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We  have  reported  on  DOD’s  acquisition  of  the  F-35  for  many  years  (see 
the  Related  GAO  Products  section  at  the  end  of  this  report)  and  more 
recently,  on  F-35  sustainment  issues.  In  September  2014,  we  found  that: 
(1)  DOD  had  not  fully  addressed  several  issues — including  ALIS-related 
problems — that  posed  risks  to  the  long-term  affordability  and  operational 
readiness  of  the  program;  (2)  DOD  had  not  linked  established 
sustainment  affordability  targets  with  military  service  budgets,  meaning 
that  the  targets  may  not  be  representative  of  what  the  services  could 
actually  afford;  and,  (3)  DOD  had  weaknesses  with  respect  to  a  few  cost 
assumptions  and  did  not  include  all  necessary  analyses  to  make  its 
sustainment  cost  estimates  fully  reliable.4  With  respect  to  ALIS,  we 
reported  that  although  DOD  had  originally  planned  to  have  a  fully 
functional  and  effective  logistics  system  by  2017 — 7  years  after  originally 
intended — to  ensure  operational  readiness  and  availability,  DOD  did  not 
have  a  performance  measurement  process,  with  metrics  and  targets,  to 
determine  and  address  performance  issues  with  ALIS.  We  recommended 
that  DOD  establish  a  performance  measurement  process  for  ALIS  that 
included,  but  was  not  limited  to,  performance  metrics  and  targets  that  are 
based  on  intended  behavior  of  the  system  in  actual  operations  and  that 
tie  system  performance  to  user  requirements.  DOD  concurred  with  our 
recommendation  and,  as  of  January  2016,  was  in  the  process  of 
developing  a  performance-measurement  process  for  ALIS. 

Citing  the  importance  of  a  fully  functional  ALIS  to  the  overall  performance 
on  the  F-35,  the  Subcommittee  on  Readiness,  House  Armed  Services 
Committee,  asked  us  to  review  the  status,  costs,  and  risks  associated 
with  ALIS.  Subsequently,  the  National  Defense  Authorization  Act  for 
Fiscal  Year  2016  included  a  provision  that  we  report  on  ALIS,  including 
an  assessment  of  the  capability  of  the  program  to  address  performance 
problems  within  planned  resources.5  This  report  assesses  the  extent  to 
which  DOD  has  (1)  a  plan  to  ensure  that  ALIS  is  fully  functional  as  key 
program  milestones  approach,  (2)  credibly  and  accurately  estimated 
costs  associated  with  ALIS,  and  (3)  developed  a  plan  to  manage  ALIS 
training  for  users. 


4GAO,  F-35  Sustainment:  Need  for  Affordable  Strategy,  Greater  Attention  to  Risks,  and 
Improved  Cost  Estimates,  GAO-14-778  (Washington,  D.C.:  Sept.  23,  2014). 

5Pub.  L.  No.  114-92,  §  242  (2015). 
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To  determine  the  extent  to  which  DOD  has  a  plan  to  ensure  ALIS  is  fully 
functional  by  key  program  milestones — which  include  the  Air  Force  and 
Navy  initial  operational  capability  declarations  and  the  start  of  full-rate 
production  for  the  F-35  program — we  reviewed  the  F-35  program’s 
sustainment  documents  and  conducted  site  visits  at  a  nongeneralizable 
sample  of  five  F-35  operational  and  training  sites:  Eglin  Air  Force  Base, 
Marine  Corps  Air  Base  Yuma,  Luke  Air  Force  Base,  Edwards  Air  Force 
Base,  and  Nellis  Air  Force  Base.  We  selected  these  sites  in  consultation 
with  service  officials  to  ensure  we  obtained  perspectives  from  primary 
operational  and  testing  sites.6  For  the  purposes  of  this  review,  we  focused 
solely  on  the  U.S.  F-35  fleet.  During  these  visits,  we  also  held  17 
nongeneralizable  focus  group  sessions  with  a  range  of  ALIS  users  (e.g., 
maintainers,  pilots)  from  all  three  services  to  obtain  information  on  the 
operability  and  deployability  of  ALIS,  and  how  any  ALIS  issues  may  pose 
risks  to  F-35  operations  and  sustainment.  These  focus  groups  also 
included  contractor  personnel  responsible  for  administering  ALIS  and 
providing  training  at  these  sites.  Approximately  a  total  of  120  people 
participated  in  these  focus  groups.  Additionally,  we  interviewed  key  DOD 
and  contractor  officials.  We  evaluated  the  information  we  obtained  for 
consistency  with  GAO’s  Schedule  Assessment  Guide  and  DOD’s  System 
Engineering  Guide  for  System  of  Systems  that  provide  guidance  and  best 
practices  on  how,  prior  to  meeting  key  milestones,  a  plan  should  be 
developed  to  address  specific  risks  that  may  be  associated  with  major 
weapon  acquisitions.7 

To  determine  the  extent  to  which  DOD  has  credibly  and  accurately 
estimated  ALIS  costs,  we  evaluated  the  reliability  of  DOD’s  estimate  of 
ALIS  costs  contained  in  the  most  recent  F-35  program  office  estimate  of 


6These  locations  included  Air  Force  and  Marine  Corps  bases.  In  addition,  although  we  did 
not  visit  a  Navy  installation,  we  spoke  with  Navy  personnel  working  in  the  F-35  program  at 
one  of  these  locations. 

7GAO,  Schedule  Assessment  Guide,  GAO-12-120G  (Washington,  D.C.:  May  2012)  and 
Department  of  Defense  Systems  Engineering  Guide  for  System  of  Systems  (Washington, 
D.C.:  August  2008). 
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F-35  operating  and  support  (O&S)  costs8  using  characteristics  contained 
in  GAO’s  Cost  Estimating  and  Assessment  Guide.9  For  the  purposes  of 
this  review,  we  conducted  a  limited  assessment  and  used  two  of  the  four 
general  characteristics  of  sound  cost  estimating  included  in  this  guide: 
being  credible  and  accurate.  We  chose  to  use  these  two  characteristics 
because  ALIS  is  not  a  stand-alone  weapon  system  program  with  its  own 
cost  estimate,  and  its  projected  costs  are  included  in  the  larger  F-35  cost 
estimate.  Therefore,  we  only  assessed  ALIS  costs  and  not  an  entire  life- 
cycle  cost  estimate,  and  determined  that  it  would  not  be  appropriate  to 
assess  whether  the  estimate  was  comprehensive  or  well-documented 
because  ALIS  costs  represent  one  element  of  the  total  F-35  cost 
estimate — less  than  2  percent  of  projected  F-35  sustainment  costs.  To 
determine  whether  the  credible  and  accurate  characteristics  were  met,  we 
reviewed  cost-estimating  documentation,  including  data  sources, 
assumptions,  and  cost  models,  and  we  interviewed  cost-estimating 
officials  from  the  F-35  program  office  and  the  Office  of  the  Director  for 
Cost  Assessment  and  Program  Evaluation  (CAPE).  We  also  interviewed 
other  officials  from  the  F-35  program  office  and  service  headquarters  to 
discuss  the  cost  effects  of  ALIS  schedule  delays  and  development  issues. 
We  found  the  data  to  not  be  fully  reliable,  which  we  discuss  in  further 
detail  in  the  report  and  in  appendix  II. 

Finally,  to  determine  the  extent  to  which  DOD  has  developed  a  plan  to 
manage  ALIS  training  for  users,  we  reviewed  key  documentation  related 
to  ALIS  and  F-35  training,  and  used  information  from  our  focus-group 
sessions  with  a  range  of  ALIS  users  from  all  three  services  to  obtain 
information  on  the  current  state  of  ALIS  training.  We  also  interviewed  key 
DOD  and  contractor  officials.  We  evaluated  all  of  the  information  we 
received  using  GAO-developed  and  industry  best  practices  for 


8The  F-35  program  office  updates  its  cost  estimate  annually.  The  most  recent  estimate 
that  was  completed,  approved,  and  promulgated  during  our  review  was  the  2014  estimate, 
released  in  spring  2015.  The  2015  estimate  is  currently  being  reviewed  within  the  program 
office  and  is  expected  to  be  finalized  after  issuance  of  this  report  in  spring  2016.  The 
Office  of  the  Director  for  Cost  Assessment  and  Program  Evaluation  (CAPE)  conducted  a 
separate  estimate  in  2013  but  has  not  updated  it  since  then.  In  September  2014,  we 
reported  on  the  reliability  of  CAPE’s  estimate.  See  GAO-14-778. 

9GAO,  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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information-technology  training,  and  guidance  on  assessing  strategic 
training  and  development  efforts  in  the  federal  government.* 1 2 3 * * * * * * 10 

We  conducted  this  performance  audit  from  April  2015  to  April  2016  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  Appendix  I  provides  a  more 
detailed  description  of  our  scope  and  methodology. 


Background 


F-35  Program  and  Costs  The  F-35  Lightning  ll  program,  also  known  as  the  Joint  Strike  Fighter,  is  a 

joint,  multinational  acquisition  intended  to  develop  and  field  a  family  of 
next-generation  strike  fighter  aircraft  for  the  United  States  Air  Force, 

Navy,  and  Marine  Corps,  and  eight  international  partners.11  According  to 
DOD,  there  will  be  three  variants  of  the  F-35: 

1 .  The  conventional  takeoff  and  landing  (CTOL)  variant,  designated  the 
F-35A,  will  be  a  multirole,  stealthy  strike  aircraft  replacement  for  the 
Air  Force’s  F-16  Falcon  and  the  A-10  Thunderbolt  II  aircraft,  and  will 
complement  the  F-22A  Raptor  (see  fig.  1). 

2.  The  short  takeoff  and  vertical  landing  (STOVL)  variant,  designated  the 
F-35B,  will  be  a  multirole,  stealthy  strike  fighter  that  will  replace  the 
Marine  Corps’  F/A-18C/D  Hornet  and  AV-8B  Harrier  aircraft. 

3.  The  carrier-suitable  variant  (CV),  designated  the  F-35C,  will  provide 
the  Navy  a  multirole,  stealthy  strike  aircraft  to  complement  the  F/A-18 


10GAO,  Information  Technology  Training:  Practices  of  Leading  Private-Sector  Companies, 

GAO-03-390  (Washington,  D.C.:  Jan.  31, 2003)  and  Human  Capital:  A  Guide  for 

Assessing  Strategic  Training  and  Development  Efforts  in  the  Federal  Government, 

GAO-04-546G  (Washington,  D.C.:  March  2004). 

11  The  international  partners  are  the  United  Kingdom,  Italy,  the  Netherlands,  Turkey, 

Canada,  Australia,  Denmark,  and  Norway.  These  nations  contributed  funds  for  system 

development  and  all  but  Canada  and  Denmark  have  signed  agreements  thus  far  to 
procure  aircraft.  In  addition,  Israel,  Japan,  and  South  Korea  have  signed  on  as  foreign 

military  sales  customers. 
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E/F  Super  Hornet.  The  Marine  Corps  will  also  field  a  limited  number  of 
F-35C  CVs. 

Lockheed  Martin  is  the  primary  aircraft  contractor  and  Pratt  &  Whitney  is 
the  engine  contractor. 


Figure  1 :  F-35A  Air  Force  Conventional  Takeoff  and  Landing  Variant  (CTOL) 


Source:  Department  of  Defense.  |  GAO-16-439 


Although  the  acquisition  costs  of  the  F-35  program  are  about  $400  billion, 
the  most  significant  cost  driver  for  the  program  is  sustainment.  The  F-35 
O&S  costs  are  those  incurred  from  the  initial  system  deployment  through 
the  end  of  system  operations,  and  include  all  costs  of  operating, 
maintaining,  and  supporting  the  fielded  system.  The  F-35  program  office 
develops  an  annual  estimate  for  the  O&S  costs  of  maintaining  and 
supporting  the  F-35  for  56  years.12  In  its  most  recent  estimate  (2014),  the 
program  office  estimates  that  it  will  cost  about  $891  billion  to  sustain  the 
entire  F-35  fleet  over  its  life  cycle.13 


Autonomic  Logistics 
Information  System 
(ALIS):  Primary  Logistics 
Tool  for  the  F-35 


The  Autonomic  Logistics  Information  System  (ALIS)  is  a  system  of 
systems  that  serves  as  the  primary  logistics  tool  to  support  F-35 
operations,  mission  planning,  and  sustainment.  ALIS  helps  maintainers 
manage  tasks  including  aircraft  health  and  diagnostics,  supply-chain 
management,  and  necessary  maintenance  events.  Lockheed  Martin  is 
the  prime  contractor  for  ALIS  and  is  responsible  for  developing  and 


12Each  aircraft  has  a  30-year  service  life,  but  the  2014  estimate  spans  56  years. 
13The  $891  billion  dollar  estimate  is  in  then-year  dollars,  adjusted  for  inflation. 
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managing  the  capabilities  of  the  system,  as  well  as  developing  training 
materials  for  ALIS  users.14 

According  to  DOD,  ALIS  will  be  co-located  with  F-35  aircraft  both  at  U.S. 
military  installations  and  in  theater  to  support  missions  and  assist  with 
maintenance  and  resource  allocation.  ALIS  consists  of  the  overarching 
system,  the  applications  housed  within  it,  and  the  some  of  the  network 
infrastructure  required  to  provide  global  integrated  and  autonomic  support 
of  the  F-35  fleet.  It  comprises  both  hardware  and  software.  The  hardware 
consists  of  three  main  components: 

•  The  Autonomic  Logistics  Operating  Unit  (ALOU):  The  ALOU  is  the 
computer  server  that  all  F-35  data  ultimately  are  sent  through  and  it 
supports  communications  with  and  between  the  government  and  the 
contractor’s  systems. 

•  The  Central  Point  of  Entry  (CPE):  The  CPE  is  configured  to  provide 
software  and  data  distribution  for  the  entire  F-35  fleet  in  the  United 
States,  enables  interoperability  with  national  (government)  systems  at 
the  country  level,  and  enables  ALIS  data  connectivity  between  bases. 
Each  international  partner  operating  F-35  aircraft  is  expected  to  have 
its  own  CPE  at  other  locations. 

•  The  Standard  Operating  Unit  (SOU):  SOUs  provide  all  ALIS 
capabilities  to  support  flying,  maintenance,  and  training.  They  also 
provide  access  to  applications  to  operate  and  sustain  the  aircraft. 

As  of  February  2016,  there  was  one  operational  ALOU  and  CPE  within 
the  United  States.  Each  F-35  operating  and  testing  site  in  the  United 
States  has  a  varying  number  of  SOUs  depending  on  the  site’s  number  of 
aircraft  and  squadrons,  and  there  are  two  versions:  SOU  VI  and  SOU  V2. 
The  main  difference  between  the  two  SOUs  is  that  SOU  V2  was  designed 
to  better  meet  participants’  deployability  requirements.  While  SOU  VI 
was  housed  in  two  1,600-pound  server  racks,  SOU  V2  was  designed  to 
have  its  components  fit  into  transit  cases  that  are  two-man  portable,  each 
weighing  approximately  200  pounds.  DOD  is  planning  to  have  at  least 
one  SOU  accompany  each  F-35  squadron.  The  services  organize  their 
squadrons  differently  but  squadron  sizes  generally  range  from  10  to  24 
aircraft.  The  F-35  Operational  Requirements  Document,  which  originated 
in  March  2000  and  contains  the  performance  and  operational  parameters 
for  the  concept  of  the  F-35,  calls  for  an  incremental  development  of  F-35 


14 

For  the  purposes  of  this  report,  Lockheed  Martin  will  be  referred  to  as  the  “contractor.” 
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capabilities  by  aircraft  software  blocks  and  phased  software  releases 
during  the  system  development  and  demonstration  phase.15  This  is 
concurrent  to  the  production  and  fielding  of  small  volumes  of  aircraft 
during  low-rate  initial  production.16  ALIS’s  software  development  is 
anticipated  to  be  completed  after  two  more  of  its  versions  are  released, 
the  first  of  which  will  support  the  Air  Force  declaring  initial  operational 
capability  in  summer  2016.  The  current  fielding  status  of  ALIS  is 
illustrated  in  figure  2,  which  also  includes  the  dates  for  when  the  next 
software  upgrade — ALIS  Version  2.0.2 — is  to  be  introduced  at  each  F-35 
site.  ALIS  3.0,  which  is  the  version  expected  to  meet  the  requirements 
defined  in  the  Operational  Requirements  Document,  is  expected  to  have 
completed  its  development  and  commence  testing  by  October  2017  in 
line  with  the  end  of  system  development  and  demonstration.17  The 
release  is  to  be  fielded  under  the  low-rate  initial  production  contract  in 
early  201 8  and  is  intended  to  be  fully  functional  by  201 9. 


15System  development  and  demonstration,  is  now  known  as  the  engineering  & 
manufacturing  development  (EMD)  phase.  The  purpose  of  the  EMD  phase  is  to  develop, 
build  and  test  a  product  to  verify  that  all  operational  and  derived  requirements  have  been 
met,  and  to  support  production  or  deployment  decisions.  Because  this  program  acquisition 
plan  was  developed  prior  to  this  change,  for  the  purposes  of  this  report  we  will  continue  to 
use  the  term  “system  development  and  demonstration.” 

16Low-rate  initial  production  establishes  the  initial  production  base  for  the  system  or 
capability  increment,  provides  an  efficient  ramp  up  to  full-rate  production,  and  maintains 
continuity  in  production  pending  operational  test  and  evaluation  completion.  For  the  F-35, 
low-rate  initial  production  is  taking  place  during  the  system  development  and 
demonstration  phase. 

1  department  of  Defense,  Joint  Strike  Fighter ;  Operational  Requirements  Document 
(Washington,  D.C.:  Aug. 19,  2008). 
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Figure  2:  Location  of  United  States  ALIS  Hardware  and  Planned  ALIS  2.0.2  Software  Upgrade  Dates 


Ogden  Depot 

Utah 

•  1  SOU  VI 

•  Proposed  software  upgrade 
August  2016 

Nellis  Air  Force  Base 

Nevada 

•  2  SOU  VI 

•  1  SOU  V2 

•  Proposed  software  upgrade 
October  2016 

Edwards  Air  Force  Base 

California 

•  4  SOU  VI 

•  Proposed  software  upgrade 
October  2016 


Marine  Corps  Air 
Station  Yuma 

Arizona 


Hill  Air  Force  Base 

Utah 

•  1  SOU  VI 

•  Proposed  software 
upgrade:  July  2016 


•  1  SOU  VI 

•  2  SOU  V2 

•  Proposed  software  upgrade: 
September  2016 


Luke  Air  Force  Base 

Arizona 

•5  SOU  VI 
•  Proposed  software 
upgrade: 

November  2016 


® 

Fort  Worth 

Texas 

SOU  V2 
aposed 
software 
upgrade: 
July  2016 


se 

Florida 

•  3  SOU  VI 

•  Proposed 
software 
upgrade: 
August  2016 


USS  Wasp 

Norfolk ,  Virginia 

•  1  SOU  VI 

•  Proposed 
software 
upgrade: 
November 
2016 


Cherry  Point  Depot 

North  Carolina 

•  1  SOU  VI 

•  Proposed  software 
upgrade: 
September  2016 

Marine  Corps  Air 
Station  Beaufort 

South  Carolina 

•  2  SOU  VI 

•  Proposed  software 
upgrade: 

August  2016 


Source:  GAO  presentation  of  Department  of  Defense  information.  I  GAO-16-439 


Notes:  As  of  January  2016,  all  locations  had  been  upgraded  to  ALIS  2.0.1. 

SOU  Version  1  is  hardware  located  at  the  squadron  level  and  is  housed  in  two  1 ,600-pound  server 
racks. 


SOU  Version  2  is  hardware  located  at  the  squadron  level  and  is  designed  to  meet  participants’ 
deployability  requirements,  The  components  fit  into  transit  cases  that  are  two-man  portable  and 
weight  approximately  200  pounds  each. 


DOD  intends  for  ALIS  software  capabilities  to  include:  operational 
planning,  maintenance,  supply-chain  management,  customer-support 
services,  training,  tech  data,  system  security,  and  external  interfaces.  As 
stated  earlier,  ALIS  is  a  system  of  systems — it  comprises  multiple 
software  applications  that  perform  specific  functions  for  F-35  maintainers, 
pilots,  supply  personnel,  and  data  analysts.  These  separate  applications 
must  be  integrated  within  the  system,  as  well  as  have  interconnectivity 
with  preexisting  “legacy”  information  systems  that  are  used  by  the 
services  for  other  weapons  system  platforms.  ALIS  applications  are  being 
developed  by  the  contractor  incrementally,  with  some  applications 
currently  more  functional  than  others.  The  F-35  program’s  original 
requirements  state  that  it  must  include  a  fully  functional  and  effective 
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logistics  system  to  ensure  operational  readiness  and  availability.  As  of 
December,  average  aircraft  availability  for  the  year — the  percentage  of  F- 
35  aircraft  capable  of  performing  missions  at  a  given  time — was  around 
50  percent,  whereas  DOD  had  expected  it  to  be  60  percent.  Figure  3 
includes  major  applications  within  ALIS,  their  intended  purpose,  and  the 
F-35  program  office’s  assessment  of  the  functionality  status  of  each 
application.  As  figure  3  shows,  according  to  DOD,  most  of  the 
applications  within  ALIS  currently  have  functionality  issues. 
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Figure  3:  Primary  ALIS  Applications  and  the  F-35  Program  Office’s  Assessment  of  Their  Functionality  Status  as  of  January 
2016 


Autonomic  Logistics  Information  Systems  (ALIS)  is  the 

primary  sustainment  tool  for  the  F-35  and  is  intended  to 
predict  maintenance  and  supply  issues,  automate  logistics 
support  processes,  and  provide  decision  aids  to  help  reduce 
life-cycle  sustainment  costs  and  improve  force  readiness. 


Computerized  Maintenance 
Management  System  (CMMS)  is  used 
to  initiate  and  track  maintenance 
actions,  schedule  of  work,  aircraft  and 
support  equipment  status,  access  the 
JTD,  and  provide  traceability  to  “as 
built”  and  “as  maintained”  configuration 
data. 


Minor  issues  as  reported  by  F-35  program  office 


Maintenance  Vehicle  Interface  (MVI) 

provides  an  interface  to  the  aircraft  for 
the  portable  maintenance  aid.  This 
allows  the  maintainer  to  operate 
aircraft  subsystems,  and  to  retrieve 
and  display  aircraft  status  including  fuel 
levels,  pump  status,  and  weapon 
status. 


Minor  issues  as  reported  by  F-35  program  office 


Propulsion  processes  the  data 
downloaded  from  the  aircraft,  including 
propulsion  Health  Reporting  Codes  as 
a  basis  for  AVD/H  to  generate  work 
orders  that  are  passed  to  CMMS. 


Minor  issues  as  reported  by  F-35  program  office 


Source:  GAO  presentation  of  Department  of  Defense  analysis.  |  GAO-16-439 


Note:  According  to  DOD  officials,  the  functionality  status  (major,  minor,  no  issues)  of  applications 
within  ALIS  is  based  on  user  feedback. 
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ALIS  was  originally  scheduled  to  be  completed  fortesting  in  2010,  as 
shown  in  figure  4.  However,  that  same  year  the  program  triggered  a 
Nunn-McCurdy  breach,18  when  unit  cost  growth  exceeded  critical 
thresholds.  As  a  result,  the  F-35  program  (including  ALIS)  was 
rebaselined  in  2012,  which  established  a  new  acquisition  baseline,  and 
Milestone  B  was  recertified  after  the  breach.19 


Figure  4:  ALIS  Schedule  History,  Original  Timeline,  and  Rebaseline 


Original  scheduled  completion  of 
Autonomic  Logistics  Information 
Systems  (ALIS)  3.0  for  testing 


2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014 


2001:  Milestone  B  is  approved  and 
Lockheed  Martin  is  selected  as  the 
prime  contractor.3 
The  program  begins  development. 


T  1 

2010:  F-35  2012:  F-35 


program 
triggers  a 
Nunn-McCurdy 
Breach1* 


program  is 
rebaselined, 
including  ALIS 


After  rebaselining,  ALIS 
3.0  is  scheduled  to  be 
completed  for  testing 


2015  |  2016  |  2017  [  2018  |  2019  \  2020-2040 


I  1  M 

2019:  Planned 
milestone  decision  to 
enter  full-rate  production 

2018:  Navy  is  scheduled 
to  declare  initial 
operational  capability 

2016:  Air  Force  is  scheduled  to  declare 
initial  operational  capability  in  August 

2015:  Marine  Corps  declares 
initial  operational  capability  in  July 


Source:  GAO  analysis  of  Department  of  Defense  information.  I  GAO-16-439 

aAs  part  of  DOD’s  acquisition  process,  Milestone  B  initiates  the  engineering  &  manufacturing 
development  phase. 

bSection  2433  of  title  10  of  the  United  States  Code,  commonly  referred  to  as  Nunn-McCurdy,  requires 
DOD  to  notify  Congress  whenever  a  major  defense  acquisition  program’s  unit  cost  experiences  cost 
growth  that  exceeds  certain  thresholds.  This  is  commonly  referred  to  as  a  Nunn-McCurdy  breach. 
Significant  breaches  occur  when  the  program  acquisition  unit  cost  or  procurement  unit  cost  increases 


1 8 

Section  2433  of  title  10  of  the  United  States  Code,  commonly  referred  to  as  Nunn- 
McCurdy,  requires  DOD  to  notify  Congress  whenever  a  major  defense  acquisition 
program’s  unit  cost  experiences  cost  growth  that  exceeds  certain  thresholds.  This  is 
commonly  referred  to  as  a  Nunn-McCurdy  breach.  Significant  breaches  occur  when  the 
program  acquisition  unit  cost  or  procurement  unit  cost  increases  by  at  least  15  percent 
over  the  current  baseline  estimate  or  at  least  30  percent  over  the  original  estimate.  For 
breaches  of  the  critical  threshold,  as  was  the  case  with  the  F-35  in  2010,  when  these  unit 
costs  increase  at  least  25  percent  over  the  current  baseline  estimate  or  at  least  50  percent 
over  the  original,  DOD  is  required  to  take  additional  steps,  including  conducting  an  in- 
depth  review  of  the  program.  Programs  with  critical  breaches  must  be  terminated  unless 
the  Secretary  of  Defense  certifies  to  certain  facts  related  to  the  program  and  takes  other 
actions,  including  restructuring  the  program.  10  U.S.C.  §  2433a. 

19As  part  of  DOD’s  acquisition  process,  Milestone  B  provides  authorization  for  the 
program  to  enter  the  engineering  &  manufacturing  development  phase. 
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by  at  least  15  percent  over  the  current  baseline  estimate  or  at  least  30  percent  over  the  original 
estimate.  For  critical  breaches,  as  was  the  case  with  the  F-35  in  2010,  when  these  unit  costs  increase 
at  least  25  percent  over  the  current  baseline  estimate  or  at  least  50  percent  over  the  original,  DOD  is 
required  to  take  additional  steps,  including  conducting  an  in-depth  review  of  the  program.  Programs 
with  critical  breaches  must  be  terminated  unless  the  Secretary  of  Defense  certifies  to  certain  facts 
related  to  the  program  and  takes  other  actions,  including  restructuring  the  program.  10  U.S.C.  § 
2433a. 

The  F-35  program  is  currently  approaching  the  end  of  the  system 
development  and  demonstration  phase,  during  which  ALIS  has  been 
developed,  built,  and  tested  to  verify  that  operational  requirements  are 
being  met.  Concurrently,  the  program  is  in  low-rate  initial  production.20  In 
addition,  DOD  and  the  services  are  in  the  process  of  declaring  major 
milestones.  Specifically,  the  Marine  Corps  declared  initial  operational 
capability  in  July  2015.  The  Air  Force  is  scheduled  to  declare  initial 
operational  capability  in  August  2016,  and  the  Navy  is  scheduled  to  do  so 
in  2018.  By  2018,  after  all  services  have  declared  their  initial  operational 
capability,  the  program,  to  include  ALIS,  is  expected  to  reach  full 
warfighting  capability.  The  F-35  program  plans  to  begin  full-rate 
production  in  2019,  and  according  to  DOD  officials,  any  additional 
modifications  or  upgrades  with  new  capabilities  to  ALIS  will  be  part  of  the 
program’s  follow-on  modernization.21  DOD  generally  requires  programs  to 
have  established  sustainment  and  support  systems,  like  ALIS,  for  the  F- 
35  by  full-rate  production.  According  to  the  F-35  Operational 
Requirements  Document,  by  full-rate  production,  all  variants  must  be  able 
to  deploy  rapidly,  sustain  high  mission  reliability,  and  sustain  a  high 
sortie-generation  rate.22 


90 

We  have  previously  reported  on  the  overlap  between  technology  development  and 
product  development  and  production  in  this  program’s  acquisition  strategy.  In  our  prior 
work,  we  have  identified  the  lack  of  knowledge  and  high  levels  of  concurrency  as  major 
drivers  in  the  significant  cost  and  schedule  growth  as  well  as  performance  shortfalls  that 
the  program  has  experienced  since  2001. 

21  Follow-on  modernization  involves  continuous  incremental  upgrades  to  the  aircraft, 
including  ALIS,  to  keep  it  affordable,  interoperable,  and  operationally  suitable  throughout 
the  life  of  the  program.  These  upgrades  will  include  common  requirements,  corrections  to 
operational  deficiencies  identified  in  operational  or  developmental  tests,  and  capabilities 
from  the  Operational  Requirements  Document. 

22Sorties,  or  flights,  are  generated  in  support  of  testing  and  operations. 
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DOD  Is  Aware  of 
Risks  to  ALIS 
Functionality  That 
Users  Have 
Identified,  but  Does 
Not  Have  a  Plan  to 
Prioritize  and  Address 
Them  as  Key 
Program  Milestones 
Approach 


ALIS  users  (pilots,  maintainers,  administrators,  and  trainers)  in  the  focus 
groups  we  held  at  five  F-35  operational  or  testing  sites  identified  some 
benefits  of  the  system.  For  example,  maintainers  and  administrators  at 
three  sites  stated  that  they  have  seen  ALIS’s  capabilities  improve  over 
time  as  the  system’s  software  has  been  upgraded.  In  addition,  pilots  and 
maintainers  at  three  sites  expressed  confidence  in  ALIS’s  future 
capabilities  as  the  system  continues  to  improve.  Maintainers  and  trainers 
at  three  sites  also  found  ALIS’s  design — which  incorporates  multiple 
functions  within  one  single  system — helpful  for  efficiently  executing  tasks, 
when  previously  they  had  to  work  in  multiple,  separate  systems.  For 
example,  trainers  at  Eglin  Air  Force  Base  stated  that,  with  legacy  aircraft, 
there  are  separate  systems  for  tasks  such  as  recording  maintenance 
work  and  ordering  parts.  With  the  F-35,  ALIS  houses  applications  for 
these  tasks  within  its  system,  which  can  be  more  convenient  to  users.  In 
addition,  maintainers  at  two  sites  stated  that,  following  a  recent  software 
upgrade,  the  system  processes  information  faster,  which  improves 
maintenance  data  input  capabilities.  Maintainers  at  three  sites  also  told  us 
that  ALIS  performs  some  tasks  better  than  legacy  systems.  Specifically, 
maintainers  at  Eglin  Air  Force  Base  and  Nellis  Air  Force  Base  explained 
that  because  ALIS  stores  information  electronically,  it  eliminates  the  need 
for  paper-based  manuals  commonly  used  on  legacy  aircraft.  For  instance, 
maintainers  at  Eglin  Air  Force  Base  noted  that  the  technical  data  that  they 
use  to  assist  in  aircraft  repair  is  now  stored  electronically  within  ALIS  and 
can  be  updated  as  necessary,  whereas  before  this  information  was 
contained  in  multiple  paper-based  manuals,  which  were  difficult  to 
efficiently  access  and  keep  up-to-date.  Most  users  we  spoke  with 


ALIS  Users  Identified 
Benefits  but  Also  Key 
Risks  to  the  System 


DOD  is  aware  of  risks  that  could  affect  ALIS  but  does  not  have  a  plan  to 
prioritize  and  address  them  in  a  holistic  manner  to  ensure  that  ALIS  is 
fully  functional  as  the  F-35  program  approaches  key  milestones — 
including  Air  Force  and  Navy  initial  operational  capability  declarations  in 
2016  and  2018,  respectively,  and  the  start  of  the  program’s  full-rate 
production  in  2019.  During  our  focus  groups,  ALIS  users  identified  some 
beneficial  aspects  of  the  system.  However,  they  also  identified  a  variety  of 
concerns,  which  may  result  in  operational  and  schedule  risks.  The 
F-35  program  office  is  aware  of  these  issues,  but  is  currently  addressing 
them  on  a  case-by-case  basis. 
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recognized  that  ALIS  is  a  system  in  development  and  stated  that  its 
immaturity  was  to  be  expected  at  this  stage  in  the  program. 

However,  during  our  focus-group  sessions,  ALIS  users  also  identified 
several  issues,  which,  if  not  addressed,  could  result  in  operational  and 
schedule  risks.  Table  1  summarizes  the  risks  reported  by  the  majority  of 
participants  in  our  17  focus  groups.  DOD  is  aware  of  these  risks  and,  as 
discussed  later,  is  addressing  risks  on  a  case-by-case  basis. 

Table  1:  ALIS  Risks  Identified  by  Users  in  GAO’s  Focus  Groups3 

Functionality  issue 

Description  of  Autonomic  Logistics  Information  System  (ALIS)  user  concerns 

Deployability  of  ALIS 

Users  are  concerned  about  ALIS’s  ability  to  deploy  in  operational  environments  because  of  the 
large  Standard  Operating  Unit  (SOU)  server  size  and  connectivity  requirements.  The  Marine 
Corps,  which  often  deploys  to  austere  locations,  declared  initial  operational  capability  in  July 

2015  without  conducting  deployability  tests  of  ALIS.  Although  a  more  deployable  version  of  ALIS 
was  put  into  operation  in  the  summer  of  2015,  the  Department  of  Defense  (DOD)  has  not  yet 
completed  comprehensive  deployability  tests. 

No  redundancy  in  its  infrastructure 

Users  are  concerned  that  ALIS’s  current  design  results  in  all  F-35  data  produced  across  the  fleet 
to  be  routed  to  the  Central  Point  of  Entry  and  then  to  the  Autonomic  Logistics  Operating  Unit, 
with  no  backup  system  or  redundancy.  If  either  of  these  fail,  it  could  take  the  entire  F-35  fleet 
offline. 

Ability  to  communicate  with  legacy 
systems 

Users  are  concerned  that  ALIS  does  not  have  much  interoperability  with  legacy  systems.  The 
services  will  continue  using  legacy  systems  for  other  weapon  systems,  and  some  of  these  data 
must  be  shared  with  ALIS.  Because  ALIS  currently  has  little  interoperability  with  legacy  systems, 
some  of  this  information  is  currently  being  tracked  manually  outside  of  ALIS,  which  is  inefficient 
and  could  potentially  result  in  data  not  being  populated  back  into  the  system. 

Action  Request13  (AR)  process  is 
inefficient  and  problematic 

Users  report  that  the  current  AR  process  does  not  provide  transparency  of  all  ARs  submitted 
across  F-35  sites,  and  places  responsibility  for  resolving  the  requests  primarily  on  the  contractor. 

Data  accuracy  and  accessibility 
issues 

Users  report  concerns  about  data  that  reside  within  ALIS,  including  errors  related  to  missing  or 
inaccurate  information  about  parts  and  not  being  able  to  extract  raw  data  to  generate  needed 
reports. 

Off-board  Mission  Support  (OMS) 
and  Training  Management  System 
(TMS)  are  immature 

Users  report  that  both  ALIS  applications  are  important  to  their  day-to-day  operations,  but  are 
immature  and  do  not  function  as  intended.  OMS  is  a  key  application  used  by  pilots  to  conduct 
mission  planning  and  debriefing.  TMS  is  used  by  pilots  and  maintainers  to  track  training 
qualifications. 

Security  concerns 

Users  report  concerns  related  to  transferring  information  between  unclassified  and  classified 
computer  servers  and  related  to  cyber  security.  A  2012  DOD  Inspector  General’s  report0  on  ALIS 
also  highlighted  some  security  issues,  including  security  accreditation  and  testing  of  hardware. 

Source:  GAO  |  GAO-16-439 

Notes: 

aWe  visited  five  operational  and  training  sites  and  convened  focus  groups  with  a  range  of  ALIS  users. 
Specifically,  we  convened  focus  group  sessions  with  pilots,  maintainers,  administrators,  and  trainers. 
Some  of  the  trainers  and  administrators  were  contractors.  These  risks  were  identified  by  ALIS  users 
across  the  range  of  groupings  at  all  five  sites.  The  views  are  not  generalizable  to  all  users. 

bAn  AR  is  a  concern  or  question  raised  by  a  customer  or  user  about  any  area  of  the  F-35  system, 
including  ALIS. 
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cDepartment  of  Defense  Inspector  General,  Audit  of  F-35  Lightning  II  Autonomic  Logistics  Information 
System  (ALIS)  (Washington,  D.C.:  Dec.  10,  2012). 


The  risks  identified  by  ALIS  users  during  our  focus  group  sessions  are 

discussed  in  more  detail  below. 

•  ALIS  may  not  be  able  to  deploy:  Pilots,  maintainers,  and 

administrators  at  three  of  the  five  sites  we  visited  are  concerned  about 
ALIS’s  ability  to  deploy  and  function  in  forward  locations.  For  example, 
users  are  concerned  about  the  large  server  size  and  connectivity 
requirements,  and  whether  the  system’s  infrastructure  can  maintain 
power  and  withstand  a  high-temperature  environment.  The  Marine 
Corps,  which  often  deploys  to  austere  locations,  did  not  conduct 
deployability  tests  prior  to  declaring  initial  operational  capability  in  July 
201 5. 23  ALIS’s  original  requirements24  did  not  include  specific 
deployability  requirements,  so  the  system’s  original  hardware  design 
consisted  of  large,  heavy  racks  of  servers.  DOD  officials  stated  that 
the  Marine  Corps  subsequently  added  specific  requirements  for  a 
deployable  system  to  meet  its  expeditionary  mission  needs.  Although 
the  more  deployable  version  of  ALIS  was  fielded  in  summer  of  201 5, 
DOD  has  yet  to  complete  comprehensive  deployability  testing.  In 
December  2015,  the  Marine  Corps  participated  in  an  exercise  at  the 
Strategic  Expeditionary  Landing  Field  near  the  Marine  Corps  Air 
Ground  Combat  Center  in  California  (also  known  as  Twentynine 
Palms)  that  included  a  short-range,  domestic  deployability  test  of  the 
system.  According  to  DOD  officials,  the  results  were  positive  in  that 
the  Marine  Corps  transported  the  system  to  Twentynine  Palms  from 
its  Yuma  base  and  set  it  up  within  2  hours;  however,  this  test  did  not 
include  long-range,  overseas,  ship-based,  or  combat  scenarios.  Air 
Force  and  Navy  officials  stated  that  they  plan  to  conduct  deployabilty 
tests  prior  to  declaring  initial  operational  capability  over  the  next  2 
years;  however,  these  officials  expressed  concerns  over  the  ability  of 


oo 

The  Marine  Corps  conducted  F-35  tests  onboard  the  USS  Wasp  prior  to  declaring 
operational  capability  in  July  2015,  including  day  and  night  carrier  missions  and 
maintenance  exercises;  however,  these  tests  did  not  include  deployability  tests  of  ALIS. 
According  to  the  Director  of  Test  and  Evaluation,  these  tests  were  not  operationally 
representative  because  of  the  heavy  use  of  contractor  support,  and  lack  of  other  types  of 
aircraft  sharing  the  flight  deck.  He  also  noted  that  this  test  used  the  original, 
nondeployable  ALIS  server. 

24Department  of  Defense,  Joint  Strike  Fighter ;  Operational  Requirements  Document. 
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ALIS  to  function  in  austere  environments  and  in  split-squadron 
situations25  that  would  require  multiple  deployable  ALIS  servers. 

•  ALIS  does  not  have  redundancy  in  its  infrastructure:  ALIS  users 
at  three  of  the  five  sites  we  visited  are  concerned  that  a  failure  in  the 
system’s  current  infrastructure  could  degrade  the  system  and  ground 
the  fleet.  Currently,  ALIS  information,  including  data  from  all  U.S. 

F-35  sites,  flows  from  the  Standard  Operating  Units  (SOU)  to  a  single 
national  Central  Point  of  Entry,  and  then  to  the  lone  Autonomic 
Logistics  Operating  Unit  (ALOU).26  This  data  flow  process  has  no 
back  up  system  for  continuity  of  operations  if  either  of  these  servers 
were  to  fail.  Specifically,  squadron  leadership  at  two  sites  expressed 
concern  about  how  the  loss  of  electricity  due  to  weather  or  other 
damaging  situations  could  adversely  affect  fleet  operations  if  either 
the  Central  Point  of  Entry  or  the  ALOU  went  offline.  DOD  officials  told 
us  that  they  recognize  this  issue  and,  for  short-term  losses  of 
connection,  ALIS  users  are  able  to  work  offline.  Program  officials  also 
said  that  they  are  in  the  early  stages  of  trying  to  procure  up  to  two 
additional  ALOUs  and  possibly  relocating  the  Central  Point  of  Entry  to 
another  F-35  site.  However,  as  of  January  2016,  DOD  officials  had 
just  begun  to  explore  this  option  and  have  not  allocated  any  resources 
to  support  the  idea. 

•  ALIS  does  not  effectively  communicate  with  legacy  aircraft 
systems:  Maintainers  and  pilots  at  three  of  the  five  sites  we  visited 
were  concerned  that  ALIS  does  not  have  much  interoperability  with 
legacy  aircraft  systems.  For  example,  while  ALIS  was  designed  to 
house  multiple  applications  within  it,  the  Air  Force,  Navy,  and  Marine 
Corps  will  continue  to  use  legacy  systems  to  operate  and  maintain 
other  weapon  systems.  The  ability  to  share  information  between  ALIS 
and  these  legacy  systems  is  vital  due  to  the  way  the  services  operate. 
In  particular,  Marine  Corps  officials  noted  that  because  the  service 
operates  with  squadrons  that  use  data  from  the  Navy’s  legacy 


25According  to  DOD  officials,  a  split-squadron  is  when  parts  of  a  squadron  deploy 
separately,  leaving  the  full  squadron  in  multiple  locations.  This,  according  to  the  same 
officials,  is  a  common  practice  as  most  full  squadrons  do  not  often  deploy  together. 

26There  is  one  Central  Point  of  Entry  for  all  U.S.  F-35  operational  squadrons  and  aircraft, 
and  another  for  all  training  squadrons  and  aircraft.  Officials  told  us  that  the  training  Central 
Point  of  Entry  does  not  provide  redundancy. 
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system27  and  ALIS,  it  would  be  beneficial  for  those  two  systems  to 
communicate  with  one  another.  DOD  officials  stated  that  the  services 
are  responsible  for  developing  the  software  interface  that  can  take 
information  from  ALIS  and  translate  it  so  that  it  can  be  communicated 
to  the  legacy  systems.  However,  due  to  the  lack  of  interoperability 
between  ALIS  and  legacy  systems,  users  are  being  forced  to  track 
data  outside  of  ALIS,  which,  according  to  maintainers,  is  inefficient 
and  could  potentially  result  in  data  not  being  populated  back  into  the 
system. 

•  Action  Request  process  is  inefficient  and  problematic: 

Maintainers  at  four  of  the  five  sites  we  visited  told  us  that  the  current 
Action  Request28  (AR)  process  does  not  allow  for  the  effective 
reporting  and  resolution  of  F-35  aircraft  and  ALIS  issues.  Personnel 
use  an  application  within  ALIS  to  submit  an  AR  about  any  F-35 
problems,  including  those  with  ALIS  itself,  to  the  contractor  for  triaging 
and  ultimate  resolution.29  However,  these  maintainers  explained  that 
the  process  is  too  heavily  controlled  by  the  contractor  and  that  users 
lack  visibility  of  ARs  submitted  from  other  F-35  sites  or  squadrons. 
Consequently,  ALIS  users  have  to  wait  for  the  contractor  to  conclude 
that  multiple  sites  are  experiencing  a  similar  issue,  instead  of  being 
able  to  identify  common  issues  across  sites  and  obtain  timely 
solutions.  In  addition,  maintainers  at  three  sites  and  administrators  at 
one  site  reported  that  recent  ALIS  software  upgrades  have  resulted  in 
the  contractor  not  receiving  ARs,  with  users  unaware  of  this  problem 
until  they  followed  up  on  the  ARs’  status.  DOD  officials  told  us  they 
are  aware  of  the  issues  surrounding  the  AR  process  and  are 
collecting  information  on  the  types  of  ARs  submitted  from  all  sites. 
They  stated  that  the  largest  types  of  ARs  are  related  to  data  quality 


27Naval  Aviation  Logistics  Command  Management  Information  System  is  an  automated 
information  system  that  provides  aviation  maintenance  and  material  management 
personnel  with  information  on  which  to  base  daily  decisions  by  the  Navy  and  the  Marine 
Corps. 

oo 

An  AR  is  a  concern  or  question  raised  by  a  customer  or  user  about  any  area  of  the  F-35 
system,  including  ALIS. 

oq 

The  submitter  of  the  AR  prioritizes  the  request  as  either  a  Category  I,  which  requires 
more  immediate  attention,  ora  Category  II,  and  the  contractor  then  prioritizes  the  requests 
to  be  addressed  by  technicians  and  engineers.  Since  the  start  of  this  process,  35,213  ARs 
have  been  submitted  of  which  34,113  have  been  closed — this  can  mean  that  the  AR  was 
either  resolved  or  dismissed.  Of  the  897  ARs  that  were  open  as  of  January  2016,  42  were 
high-priority  Category  I  ARs. 
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and  integration  management,  and  that  this  has  been  the  case  for  the 
last  12  months. 

•  ALIS  has  data  accuracy  and  accessibility  issues:  ALIS  users  at  all 
five  sites  we  visited  are  concerned  with  data  accuracy  issues  within 
the  system,  including  missing  or  inaccurate  data  and  inaccessibility  of 
raw  data  within  ALIS.  Specifically,  maintainers  frequently  have  to 
resolve  error  messages  for  parts  linked  to  electronic  equipment  logs30 
that  contain  missing  or  inaccurate  data  when  they  try  to  fix  a  problem 
on  the  aircraft.  Maintainers  at  two  sites  stated  that  recurring  issues 
with  electronic  equipment  logs  have  caused  them  to  spend  significant 
time  resolving  these  issues  instead  of  tending  to  other  aircraft 
issues.31  Additionally,  they  stated  that  parts  requiring  scheduled 
maintenance  are  not  being  tracked  or  updated  accurately  in  ALIS. 
Program  officials  stated  that  these  are  life-limited  parts  that  must  be 
replaced  by  a  certain  time  frame  to  avoid  safety  risks  to  the  aircraft. 
To  mitigate  this  issue,  maintainers  are  currently  logging  this 
information  outside  of  ALIS.  Maintainers  at  Eglin  Air  Force  Base  said 
that  they  are  spending  13  hours  on  average  every  day  to  track  this 
information.  Finally,  maintainers  at  three  sites  stated  they  would  like 
the  ability  to  access  raw  data  in  ALIS  to  produce  service-related 
reports.  DOD  officials  stated  that  ALIS  was  designed  to  be  used 
across  services  and,  as  such,  reporting  tools  are  not  necessarily 
service-specific.  However,  ALIS  users  that  operate  the  system  daily 
continue  to  have  issues  with  accessing  the  data  required  to  keep 
aircraft  mission-capable  and  generating  service-specific  reports  for 
their  squadrons. 

•  Off-Board  Mission  Support  and  Training  Management  System 
applications  are  immature:  Pilots  and  maintainers  across  all  five 
sites  we  visited  are  concerned  with  the  maturity  and  functionality  of 
ALIS’s  Off-Board  Mission  Support  (OMS)  and  Training  Management 
System  (TMS)  applications.  OMS  is  a  key  application  designed  for 


30Electronic  equipment  logs  are  linked  to  certain  parts  on  the  F-35.  They  contain  critical 
information  about  the  parts,  such  as  date  of  manufacture  and  serial  numbers.  According  to 
DOD  officials,  this  information  is  supposed  to  be  entered  at  the  production  facility  and 
DOD  is  aware  of  these  data  inaccuracies. 

31  In  an  attempt  to  address  this  issue,  DOD  officials  told  us  that  the  Defense  Contract 
Management  Agency  recently  informed  the  contractor  about  the  recurring  errors. 
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pilots  to  conduct  essential  mission  planning  and  debriefing.32 
Specifically,  pilots  at  all  five  sites  thought  that  the  OMS  application 
was  poorly  designed,  cumbersome,  and  not  user-friendly,  especially 
with  providing  the  necessary  information  they  need  to  conduct  their 
missions.  Due  to  OMS’s  current  lack  of  functionality,  pilots  at  two 
locations  stated  that  they  are  forced  to  track  vital  mission  planning 
information,  information  expected  to  reside  within  ALIS,  outside  of  the 
system.  According  to  the  Office  of  the  Director  of  Test  and  Evaluation, 
OMS’s  lack  of  functionality  could  have  an  effect  on  combat  missions 
and  operational  tempo.33 

The  Training  Management  System  (TMS)  is  designed  for  pilots  and 
maintainers  to  track  training  qualifications  and  assign  personnel  to 
carry  out  specific  tasks  based  on  their  qualifications.  However,  pilots 
and  maintainers  at  four  sites  told  us  that  TMS  is  immature  and  does 
not  function  as  intended.  Maintainers  at  one  site  explained  that  TMS 
is  supposed  to  keep  track  of  maintainers’  and  pilots’  qualifications 
and,  based  on  that  information,  assign  proper  permission  levels  and 
controls  to  a  qualified  maintainer  to  repair  a  problem  on  the  aircraft. 
Instead,  this  is  currently  being  tracked  outside  of  ALIS,  which  is 
inefficient  and  could  potentially  result  in  this  information  not  being 
populated  back  into  the  system. 

•  Security  risks  exist:  ALIS  users  cited  concerns  related  to  the 
system’s  security.  For  example,  pilots  at  one  location  explained  that 
compact  discs  have  to  be  recorded  to  move  classified  information 
from  the  aircraft  into  the  classified  network,  rather  than  the  system 
transmitting  the  information  automatically — a  practice  that  they  said 
poses  security  risks.  In  addition,  the  ALOU  and  Central  Point  of  Entry, 
as  discussed  earlier,  are  potential  single  points  of  failure  and  could  be 
a  security  risk.  A  2012  DOD  Inspector  General’s  report34  on  ALIS  also 
highlighted  some  security  issues,  including  security  accreditation  and 
testing  of  hardware.  Since  that  report,  the  F-35  program  office  has 
formed  a  team  and  developed  a  process  to  test,  validate,  and 
continuously  monitor  the  security  of  ALIS  applications  and  their 


op 

Mission  planning  supports  simple  training  to  complex  combat  scenarios  and  includes 
data  such  as  navigation,  threats,  and  weapons. 

33Operational  tempo  is  the  rate  of  military  actions  or  missions. 

34Department  of  Defense  Inspector  General,  Audit  of  F-35  Lightning  II  Autonomic 
Logistics  Information  System  (ALIS). 
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interfaces  with  both  military  information  networks  and  the  contractor’s 
ALIS  architecture. 


DOD  Is  Addressing  ALIS 
Risks  on  a  Case-by-Case 
Basis  but  Does  Not  Have 
a  Plan  to  Prioritize  and 
Address  Them  in  a  Holistic 
Manner 


DOD  is  aware  of  the  risks  identified  by  ALIS  users,  as  well  as  others,  and 
is  addressing  some  on  a  case-by-case  basis.  However,  DOD  officials 
acknowledged  that  the  department  does  not  have  a  plan  that  would 
prioritize  and  address  key  risks  in  a  holistic  manner  as  program 
milestones  approach.  In  recent  years,  the  F-35  program  has  emphasized 
the  criticality  of  ALIS  to  the  success  of  the  F-35.  In  October  201 5,  the 
F-35  Program  Executive  Officer  stated  that  ALIS  is  a  crucial  component 
of  the  F-35  and  should  be  treated  as  its  own  weapon  system.  Additionally, 
the  Program  Executive  Officer  stated  that  the  program  office  changed  its 
organizational  structure  to  provide  more  senior  leadership  oversight  of 
ALIS.  Although  more  focus  has  been  given  to  ALIS,  according  to  DOD 
officials,  the  F-35  Program  Executive  Officer  has  reiterated  that  the  focus 
is  on  completing  the  current  development  and  testing  of  ALIS  within  the 
already  established  time  frames,  and  with  the  previously  planned  funding. 
As  a  result,  this  has  created  an  environment  of  competing  priorities  and 
limited  resources  for  the  entire  program  in  the  near  term,  including  ALIS. 


ALIS  users  have  identified  key  risks  to  ALIS’s  functionality  that  we 
highlight  in  this  report,  and  program  office  officials  acknowledge  that 
others  may  exist  as  well.  The  F-35  program  office  has  taken  some  actions 
in  an  attempt  to  address  smaller  ALIS  functionality  issues  between  major 
software  upgrades,  and  is  considering  the  procurement  of  additional  ALIS 
servers  to  add  redundancy  to  the  system.  However,  the  current  approach 
does  not  prioritize  issues  in  a  way  that  clearly  designates  which  issues 
must  be  addressed  within  the  time  left  in  the  system  development  and 
demonstration  phase,  and  which  issues  could  be  addressed  later  as  part 
of  follow-on  modernization.  GAO  guidance  and  DOD  best  practices 
emphasize  that,  prior  to  meeting  key  milestones,  a  plan  to  address 
specific  risks  that  may  be  associated  with  major  weapon  acquisitions 
should  be  developed.  Specifically,  GAO’s  Schedule  Assessment  Guide 
states  that  a  high-quality  and  reliable  acquisition  schedule  includes  the 
need  to  plan  for  and  address  major  risks  prior  to  meeting  milestone 
dates.35  DOD’s  System  Engineering  Guide  for  System  of  Systems 
includes  risk  management  as  a  key  aspect  of  system  engineering.  It  helps 
to  ensure  that  program  costs,  schedules,  and  performance  objectives  are 


35GAO-12-120G. 
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achieved  at  every  stage  in  the  life  cycle  and  helps  to  communicate  to  all 
stakeholders  the  process  for  uncovering,  determining  the  scope  of,  and 
managing  program  uncertainties.36  Although  key  milestones — such  as  Air 
Force  and  Navy  initial  operational  capability  declarations  and  the  start  of 
full-rate  production — are  quickly  approaching,  DOD  does  not  have  a  plan 
that  prioritizes  ALIS  risks  to  ensure  that  the  most  important  are 
expediently  addressed  and  that  ALIS  is  fully  functional  by  these 
milestones.  Furthermore,  by  continuing  to  address  issues  on  a  case-by- 
case  basis,  DOD  risks  that  its  solution  to  one  issue  could  exacerbate 
another — for  example,  in  addressing  a  security  risk  in  isolation,  DOD 
could  inadvertently  create  further  risks  to  data  accessibility.  According  to 
F-35  program  officials,  a  functional  ALIS  is  key  to  the  operational 
capability  of  the  aircraft  and  the  day-to-day  ability  to  sustain  the  aircraft. 
Moreover,  the  department  expects  to  significantly  increase  aircraft 
production  within  the  next  5  years,  so  the  number  of  aircraft  that  must  be 
maintained  and  kept  ready  for  flight  will  soon  grow.  By  continuing  to 
respond  to  issues  on  a  case-by-case  basis  rather  than  in  a  holistic 
manner,  there  is  no  guarantee  that  DOD  will  address  the  highest  risks, 
and  as  a  result,  DOD  may  encounter  further  schedule  and  development 
delays,  including  system  upgrades,  which  could  affect  operations  and 
potentially  lead  to  cost  increases. 


DOD  Has  Estimated 
the  Costs  of  ALIS,  but 
Including  Additional 
Analyses  and 
Information  Would 
Increase  the 
Credibility  and 
Accuracy  of  Its 
Estimate 


DOD  has  estimated  total  ALIS  costs  to  be  approximately  $16.7  billion 
over  its  56-year  life  cycle.  However,  the  estimate  is  not  fully  credible  since 
DOD  has  not  performed  uncertainty  and  sensitivity  analyses  as  part  of  its 
cost-estimating  process.  Moreover,  while  DOD  has  updated  its  estimate 
to  be  reflective  of  some  program  changes,  it  is  also  not  fully  accurate 
since  DOD  did  not  use  historical  cost  data — both  actual  data  from  ALIS 
and  data  from  comparable  programs — when  developing  its  ALIS 
estimate.  Finally,  other  costs  such  as  service  customizations  of  ALIS  may 
require  additional  future  resources,  and  manual  workarounds  to  the 
system  currently  require  additional  labor  resources. 


36Department  of  Defense,  Systems  Engineering  Guide  for  System  of  Systems. 
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DOD  Estimates  that  Total 
ALIS  Costs  Are  about 
$16.7  Billion  over  a  56- 
Year  Life  Cycle 


DOD  estimates  that  total  ALIS  costs  are  about  $16.7  billion — about  $562 
million  to  develop  the  system,  about  $1.1  billion  to  procure  hardware  and 
spare  parts,  and  about  $15.1  billion  to  sustain  it  in  then-year  dollars.37 
DOD  had  expended  approximately  $505  million  to  develop  ALIS  as  of 
December  2015,  and  the  department  estimates  that  continued 
development  will  cost  an  additional  $57  million  through  2017,  which  is 
when  DOD  expects  ALIS  3.0  will  be  released  for  testing.  In  addition  to  the 
purchase  cost  for  ALIS,  DOD  estimates  that  ALIS  will  cost  about  $15.1 
billion  to  sustain  over  a  56-year  life  cycle.  Program  officials  told  us  that 
ALIS  development  will  be  completed  within  the  planned  resources,  but 
that  the  system  will  require  follow-on  modernization  and  that  the  program 
office  is  currently  planning  for  those  additional  costs.  Table  2  provides 
more  detail  on  ALIS  cost  elements. 


Table  2:  Autonomic  Logistics  Information  System  (ALIS)  Costs 

Cost  element 

Then-year  dollars  in 
millions 

Development  costs 

Expended  as  of  December  201 5 

505 

Estimated  development  costs  remaining  through  2017 

57 

Subtotal 

$562 

Estimated  procurement  costs 

Hardware 

931 

Spare  parts 

147 

Subtotal 

$1,078 

Estimated  sustainment  costs 

Contractor  support3 

8,050 

Technology  refresh3 

3,850 

Hardware  maintenance  agreements 

1,603 

Software  licensing  agreements 

1,598 

Subtotal 

$15,101 

Total 

$16,741 

Source:  GAO  analysis  of  F-35  program  office  data.  |  GAO-16-439 


37DOD’s  estimate  of  ALIS  sustainment  costs  includes  operations  and  support  costs — the 
costs  of  personnel,  equipment,  supplies,  software,  and  services  associated  with  operating, 
modifying,  maintaining,  supplying,  and  otherwise  supporting  the  system. 
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Notes:  Procurement  and  sustainment  cost  estimates  are  from  the  F-35  program  office  2014  cost 
estimate  and  span  56  years,  from  2009  to  2065.  Figures  in  table  may  not  sum  due  to  rounding. 
Development  costs  include  both  DOD’s  and  international  partners’  share  of  developing  ALIS. 
Procurement  and  sustainment  costs  represent  only  DOD’s  share  of  estimated  ALIS  costs  and  do  not 
include  international  partners’  estimated  ALIS  costs. 

includes  costs  for  ALIS  administration  and  database  maintenance  as  well  as  manpower  for  hardware 
replacement. 

bDOD’s  definition  of  technology  refresh  is  the  periodic  replacement  of  both  custom-built  and 
commercial-off-the-shelf  system  components,  within  a  larger  DOD  weapon  system,  to  ensure 
continued  supportability  throughout  the  weapon  system’s  life  cycle. 

DOD’s  Sustainment  Cost 
Estimate  for  ALIS  Is  Not 
Fully  Credible 

We  found  that  the  program  office’s  estimate  for  ALIS  sustainment  costs 
minimally  met  the  best  practices38  for  a  credible  cost  estimate  largely 
because  it  does  not  include  uncertainty  and  sensitivity  analyses  (see  app. 

II  for  more  information  on  our  assessment  of  the  ALIS  cost  estimate). 
Every  cost  estimate  contains  a  degree  of  uncertainty  because  of  the 
many  assumptions  that  must  be  made  about  the  future.  To  mitigate  this 
uncertainty,  a  variety  of  checks  and  analyses  can  be  conducted  to 
determine  the  credibility  of  the  assumptions  and  the  estimate  as  a  whole. 
According  to  GAO’s  Cost  Estimating  and  Assessment  Guide,  cost 
estimates  should  include  uncertainty  analyses  to  determine  the  level  of 
uncertainty  associated  with  the  estimate  in  order  to  be  credible.  A 
quantitative  uncertainty  analysis  can  provide  a  broad  overall  assessment 
of  the  risk  in  the  cost  estimate.  In  addition,  credible  cost  estimates  should 
include  sensitivity  analyses  to  examine  how  changes  to  individual 
assumptions  and  inputs  affect  the  estimate  as  a  whole. 

Although  ALIS  is  a  multibillion  dollar  system,  it  is  not  a  formally 
designated  stand-alone  weapons  system  program;  therefore,  DOD  is  not 
required  to  perform  a  separate  estimate  of  the  system’s  projected  costs. 
ALIS  is  one  cost  element  within  the  overall  F-35  program,  and  the 
program  office  estimates  that  the  system  will  constitute  less  than  2 
percent  of  the  $891.1  billion  total  sustainment  costs  of  the  program.  Cost 
estimators  told  us  that  despite  its  critical  importance  in  operating  and 
maintaining  the  entire  F-35  fleet,  since  ALIS  constitutes  a  small  portion  of 
the  total  estimate,  it  is  not  considered  a  major  cost  driver  of  the  program. 
Therefore,  cost  estimators  told  us  DOD’s  cost-estimating  guidance  does 
not  require  them  to  perform  uncertainty  or  sensitivity  analyses  for  ALIS. 

As  part  of  the  planning  process  for  F-35  sustainment,  program  officials 

38These  best  practices  are  found  in  GAO’s  Cost  Estimating  and  Assessment  Guide.  See 
GAO-09-3SP 
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have  examined  the  role  of  F-35  information  systems  (which  includes 
ALIS)  in  relation  to  five  major  value  streams — maintenance,  supply  chain, 
training,  management,  and  sustaining  engineering.  While  program 
officials  said  this  process  helped  map  the  connections  and 
interdependence  between  program  elements,  it  did  not  quantify  in 
uncertainty  or  sensitivity  analyses  the  potential  effects  of  ALIS  on 
sustainment  costs  specifically. 

Additionally,  DOD  did  not  perform  analyses  to  determine  how  further  ALIS 
schedule  delays  or  functionality  issues  could  affect  other  F-35  costs.  In 
lieu  of  the  analyses,  program  officials  stated  that  they  assume  that  if  ALIS 
does  not  perform  as  planned,  aircraft  could  not  be  flown  as  frequently  as 
intended,  and  this  lower-than-expected  utilization  rate  would  therefore 
decrease  sustainment  costs.  A  2013  DOD-commissioned  study  on 
reducing  F-35  costs  found  areas  of  potential  cost  savings,  but  the 
implementation  plan  for  this  study  also  found  that  ALIS  presents 
significant  risk  to  the  program.  Particularly,  this  plan  found  that  any 
functionality  problems  or  schedule  slippage  with  ALIS  will  have  a 
significant  impact  on  costs — with  downstream  additional  costs  due  to 
performance  and  schedule  delays  potentially  reaching  up  to  $20-100 
billion.39  The  different  conclusions  drawn  by  the  program  office  and  this 
study  suggest  that  DOD  could  better  understand  the  effects  that  ALIS 
issues  could  have  on  overall  program  costs.  Without  performing 
uncertainty  or  sensitivity  analyses,  DOD  will  not  understand  how 
variabilities  in  ALIS-related  assumptions  could  affect  the  estimate  as  a 
whole  and  the  potential  range  of  costs  resulting  from  these  variabilities. 


DOD’s  ALIS  Cost  Estimate 
Is  Not  Fully  Accurate,  and 
Upcoming  Program 
Changes  May  Affect 
Future  Estimates 


We  also  assessed  the  ALIS  estimate  for  accuracy  and  found  that  DOD 
partially  met  the  standards  for  an  accurate  cost  estimate.  GAO’s  Cost 
Estimating  and  Assessment  Guide  states  a  cost  estimate  should  be 
based  on  historical  data — both  actual  costs  of  the  program  and  those  of 
comparable  programs — which  can  be  used  to  challenge  optimistic 
assumptions  and  bring  more  realism  to  a  cost  estimate.  While  we  found 
that  the  DOD  substantially  met  some  best  practices  for  an  accurate  cost 
estimate  by  properly  adjusting  for  inflation  and  not  including  mathematical 
errors,  the  estimate  uses  contractor-provided  data  for  material  costs 


39McKinsey  &  Company,  F-35  O&S  Cost  Reduction  Strategy  (Sept.  17,  2013);  and,  JSF 
O&S  Cost  Reduction  Implementation  Plan  (Sept  20,  2013). 
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instead  of  actual  ALIS  costs  or  historical  cost  data  from  analogous 
programs  that  would  make  the  estimate  more  accurate.  Cost  estimating 
officials  said  that  they  did  not  base  their  ALIS  estimates  on  historical  cost 
data  because  they  believe  that  there  are  no  programs  analogous  to  ALIS. 
For  example,  there  is  a  logistics  system  for  the  Air  Force’s  F-22 
program — also  a  fifth-generation  aircraft — but  officials  stated  that  is  far 
less  complex  than  ALIS  and  does  not  include  all  of  ALIS’s  applications 
and  intended  functions.40  However,  multiple  versions  of  ALIS  have  been 
fielded  since  2010  and  using  historical  data  on  known  ALIS  costs,  as  well 
as  analogous  data  from  the  F-22  or  other  programs,  would  make  the 
estimate  more  accurately  representative  of  likely  sustainment  costs  (see 
app.  II  for  more  information  on  our  assessment). 

GAO’s  Cost  Estimating  and  Assessment  Guide  also  states  that  an 
estimate  should  be  updated  regularly  to  reflect  significant  changes  in  the 
program — such  as  when  schedules  or  other  assumptions  change — so 
that  it  is  always  reflecting  current  status.  The  program  office  updates  its 
estimate  annually  and  incorporates  program  changes  and  evolving 
assumptions  in  these  updates,  documenting  the  changes  from  year  to 
year.  However,  the  program  office  does  not  update  all  elements  of  the 
cost  estimate.  For  example,  technology  refresh  accounts  for 
approximately  25  percent  of  ALIS  sustainment  costs  and  program  officials 
were  able  to  tell  us  the  assumptions  used  to  calculate  these  costs, 
however,  they  were  unable  to  tell  us,  or  identify  within  the  estimating 
model,  where  these  data  came  from  or  when  their  underlying 
assumptions  were  developed.41  Additionally,  over  the  course  of  our 
review,  program  officials  highlighted  some  recent  or  upcoming  program 
changes,  such  as  the  need  for  additional  infrastructure,  that  were  not 
included  in  their  last  estimate. 


40GAO’s  Cost  Estimating  and  Assessment  Guide  states  that  cost  estimators  should  collect 
actual  cost  data  from  a  list  of  similar  and  legacy  programs.  Since  most  new  programs  are 
improvements  of  existing  ones,  data  should  be  available  that  share  common 
characteristics  with  the  new  program.  The  analogy  cost  estimating  method  uses  actual 
costs  from  a  similar  program  with  adjustments  to  account  for  differences.  A  cost  estimator 
typically  uses  this  method  when  there  is  little  actual  cost  data  available  but  the  technical 
and  program  definition  is  good  enough  to  make  the  necessary  adjustments  and 
comparisons  to  a  predecessor  system. 

41 

DOD's  definition  of  technology  refresh  is  the  periodic  replacement  of  both  custom-built 
and  commercial-off-the-shelf  system  components,  within  a  larger  DOD  weapon  system,  to 
ensure  continued  supportability  throughout  the  weapon  system’s  life  cycle. 
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The  program  office  and  services  are  exploring  ways  to  decrease  F-35 
sustainment  costs.  For  example,  the  services  are  taking  different 
approaches  to  administering  and  maintaining  ALIS,  with  the  Marine  Corps 
planning  on  using  its  own  personnel,  and  the  Air  Force  planning  on  using 
contractor  support  to  administer  the  system,  troubleshoot  problems,  and 
keep  the  system  operating  smoothly  with  software  patches  and  other 
continuous  improvements.  Based  on  these  service  plans,  the  program 
office  estimates  that  ALIS  administration  will  cost  more  than  $7  billion 
over  the  F-35  life  cycle,  in  addition  to  about  $1  billion  for  other  contractor 
labor  needs.  Program  officials  said  that  these  amounts  may  change 
based  on  upcoming  sustainment  decisions,  including  a  potential  way  to 
decrease  administrative  costs  by  establishing  regional  centers  that  could 
provide  this  support  to  a  number  of  F-35  sites  rather  than  having 
contracted  administrators  at  each  site. 

Other  program  changes  have  the  potential  to  increase  ALIS  costs  and 
future  estimates.  Although  DOD  has  included  estimated  costs  of  ALIS 
technology  refresh  and  software  licensing,  program  officials  have  stated 
that  the  current  estimate  does  not  sufficiently  capture  the  full  costs  of 
follow-on  modernization  that  will  be  required  when  upgrades  or  new 
versions  of  the  system’s  commercial  off-the-shelf  software  are  released. 
Because  ALIS  comprises  multiple  applications  and  interfaces  with  service 
networks  and  legacy  information  systems,  engineers  will  have  to 
reintegrate  ALIS  with  other  systems  as  well  as  the  applications  within 
ALIS  in  step  with  continuous  upgrades  and  software  improvements. 
Although  DOD  expects  ALIS  3.0  to  be  the  fully  capable  version  that 
meets  program  requirements,  officials  at  the  program  office  and  from  all 
three  services  stressed  that  ALIS  will  not  be  a  static  system  and  that 
improvements  and  upgrades  will  not  only  be  expected,  but  required,  to 
keep  ahead  of  technology  obsolescence  and  evolve  with  emerging  threat 
environments.  Program  officials  stated  that  there  is  a  need  to  build  these 
follow-on  modernization  costs  into  the  budgeting  and  cost-estimating 
processes. 

Program  officials  also  stated  that  they  may  procure  up  to  two  more 
ALOUs  for  back-up  and  necessary  redundancy  and  that,  in  addition  to 
addressing  the  current  risk  of  the  single  ALOU  becoming  a  single  point  of 
failure,  these  additional  ALOUs  may  facilitate  greater  government 
operation  of  ALIS  and  increase  the  potential  for  greater  competition  of 
future  sustainment  contracts.  The  program  office  bases  much  of  its 
estimate  on  inputs  from  the  services,  such  as  their  expected  personnel 
needs  and  how  they  plan  to  operate  and  sustain  their  F-35  squadrons. 
The  program  plans  to  field  one  SOU  for  each  F-35  squadron,  and  another 
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SOU  at  each  forward  operating  location.  Since  deploying  squadrons  do 
not  normally  transit  to  the  forward  location  together  as  one  unit  but  rather 
in  a  more  staggered  fashion,  having  an  SOU  at  both  the  squadron’s  home 
base  and  another  prepositioned  at  the  forward  deployed  location  would 
avoid  potential  problems  of  having  a  split  squadron  sustained  with  just 
one  SOU.  Having  additional  SOUs  positioned  at  forward  deployed 
locations  would  likely  increase  procurement  costs  and  the  downstream 
costs  of  maintaining  and  replacing  them. 

While  the  program  office  has  been  incorporating  some  program  changes 
and  adjusting  its  cost  estimating  assumptions  as  the  F-35  program  grows 
and  evolves,  it  is  important  to  continue  this  effort  in  its  annual  estimate 
updates  to  reflect  current  or  planned  program  changes  such  as  those 
described  above.  Additionally,  as  the  program  gains  more  experience 
operating  ALIS,  using  historical  data — especially  actual  program  costs  as 
they  become  available — as  the  basis  for  its  estimates  can  result  in  a  more 
accurate  and  realistic  picture  of  ALIS  costs.  Unless  DOD’s  estimate  is 
based  on  an  assessment  of  the  most  likely  costs,  the  estimate  may  not  be 
representative  of  how  much  it  will  cost  to  sustain  ALIS  and  may  inhibit 
informed  decision  making. 


Service  Customizations 
and  Workarounds  Used  to 
Overcome  ALIS 
Limitations  Could  Require 
Additional  Resources 


There  are  other  ALIS-related  costs  that  are  not  included  in  the  estimate 
but  that  may  place  additional  financial  or  manpower  burdens  on  the 
services  as  ALIS  continues  to  be  concurrently  developed  and  fielded. 
Services  are  responsible  for  bearing  the  costs  of  engineering  any  service- 
specific  ALIS  customizations.  According  to  DOD  officials,  ALIS 
engineering  changes  must  be  agreed-upon  by  all  services  and  partners  in 
order  to  become  new  requirements.  The  program  office  then 
communicates  these  requested  changes  to  the  contractor  software 
engineers  developing  ALIS.  However,  as  ALIS  users  become  more 
familiar  with  the  system  and  its  limitations,  the  services  may  request 
additional  changes  to  the  design  of  ALIS  to  meet  their  specific 
airworthiness  and  reporting  requirements,  and  incur  the  additional  costs 
to  meet  these  needs. 


Some  service-specific  reporting  requirements  are  being  met  through  the 
use  of  time-consuming  workarounds  employed  by  ALIS  users  to 
compensate  for  current  system  limitations.  According  to  program  officials, 
some  service-specific  reporting  requirements  are  not  addressed  by 
current  ALIS  functionality.  For  example,  maintainers  at  several  sites  said 
that  they  must  request  maintenance  data  entered  and  housed  within  ALIS 
from  the  contractor  because  they  cannot  access  this  information 
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independently.  They  then  use  these  raw  data  to  generate  reports  on 
aircraft  status  using  software  programs  outside  of  ALIS  because  the 
system  does  not  have  the  capability  to  extract  and  process  the  data  in  the 
form  that  the  services  require.  Program  officials  told  us  that  they  expect 
some  of  these  workarounds  to  become  unnecessary  after  upcoming 
system  improvements.  Other  ALIS  functionality  issues  that  create  the 
need  for  workarounds  do  not  have  scheduled  solutions,  so  both  program 
and  service  headquarters  officials  are  in  the  process  of  examining  the  use 
of  these  workarounds  to  determine  whether  they  are  truly  necessary  and 
will  require  an  ALIS  design  change  or  whether  service  expectations 
should  be  better  managed.  Program  officials  told  us  that  this  workaround 
is  not  a  problem  with  ALIS  itself,  but  more  an  issue  of  the  services 
requiring  reports  that  the  system  was  not  designed  to  provide.  Users  at  all 
the  sites  we  visited  told  us  that  ALIS  should  have  the  functionality  to 
create  reports  that  they  need. 

Program  officials  said  that  there  is  not  an  extra  financial  cost  associated 
with  these  workarounds  since  they  are  performed  by  service  personnel, 
but  rather  there  is  an  opportunity  cost  to  performing  them — the  additional 
time  personnel  spend  on  manually  analyzing  data  and  generating  reports 
rather  than  performing  their  primary  job  duties.  Neither  the  program  office 
nor  the  services  track  the  use  of  ALIS  workarounds  across  all  F-35 
squadrons,  but  squadron  leadership  at  Eglin  Air  Force  Base  provided 
examples  of  workarounds  performed  at  their  site  including  the  estimated 
amount  of  time  personnel  must  spend  to  overcome  ALIS  shortcomings. 
They  estimate  that  personnel  at  that  location  alone  spend  approximately 
150  man-hours  per  week  on  these  workarounds — nearly  the  equivalent  of 
having  four  full-time  personnel  dedicated  to  manually  creating  reports 
because,  according  to  officials,  ALIS  currently  cannot.  Eglin  Air  Force 
Base  personnel  have  created  and  updated  14  manual  products  mainly 
because  of  ALIS  report  limitations  and  because  ALIS  does  not  track 
information  needed  by  senior  leaders  for  unit-level  analysis  and 
subsequent  maintenance  and  logistics  decisions.  They  added  that  without 
these  manual  products,  analysis,  scheduling,  and  maintenance 
operations  would  suffer  degradation  and  that  mission  accomplishment 
could  be  jeopardized. 

The  use  of  workarounds  varied  across  the  F-35  sites  we  visited,  and 
officials  said  that  Eglin  Air  Force  Base’s  examples  are  not  necessarily 
representative  of  other  sites’  workarounds.  They  provide  a  sense  of  scale, 
however,  for  the  labor  burden  placed  on  personnel  because  of  current 
system  immaturity  and  functionality  issues.  Personnel  at  F-35  sites  we 
visited  told  us  that  the  extra  time  they  spend  on  ALIS  workarounds 
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detracts  from  the  time  they  have  to  maintain  proficiency  in  their 
specialties,  prevents  them  from  coaching  and  training  their  subordinates, 
and  decreases  the  amount  of  time  they  have  to  perform  collateral  duties. 
They  added  that  ALIS  is  not  yet  fully  autonomic  and  will  require  significant 
additional  system  improvements  for  it  to  perform  to  their  expectations. 

The  use  of  these  ALIS  workarounds  highlights  the  current  immature  state 
of  the  system  and  underscores  the  need  for  DOD  to  prioritize,  and 
address  key  ALIS  risks  as  discussed  earlier  in  this  report.  As  the  F-35 
program  increases  production  over  the  next  several  years,  sustained 
attention  to  addressing  these  issues  and  improving  estimates  of  ALIS 
costs  can  help  decision  makers  better  direct  resources  and  ensure  that 
ALIS  meets  the  needs  of  its  users. 


DOD  Does  Not  Have 
a  Program-Wide 
Training  Plan  for  ALIS 
Users  but  Has  Taken 
Some  Initial  Steps  to 
Address  Training 
Shortfalls 


DOD  does  not  have  a  program-wide  training  plan  for  ALIS,  but  has  taken 
initial  steps  to  address  training  shortfalls.  According  to  ALIS  users  at  all 
five  F-35  operational  and  training  sites  where  we  conducted  our  focus 
groups,  training  for  ALIS  is  ineffective,  and  lacks  a  standardized,  common 
curriculum  for  teaching  users  how  to  operate  ALIS.  Basic  ALIS  training 
courses  are  made  available  to  all  ALIS  users  at  the  F-35  Academic 
Training  Center  at  Eglin  Air  Force  Base.  The  training,  according  to  ALIS 
users  across  all  five  sites,  consists  of  a  series  of  PowerPoint  slides  that 
are  geared  toward  illustrating  the  conceptual  nature  of  the  system — 
showing  how  all  the  applications  within  ALIS  are  supposed  to  work — 
rather  than  how  to  actually  operate  the  system  as  it  currently  exists. 
Furthermore,  several  ALIS  users  described  the  training  as  ineffective 
because  it  does  not  teach  users  how  to  explicitly  operate  specific  ALIS 
applications.  For  example,  the  slideshow-based  classroom  training,  which 
is  optional  for  ALIS  users,  is  not  customized  for  the  different  users  of 
ALIS.  Pilots,  maintainers,  supply  personnel,  data  analysts,  and  other  F-35 
specialists  are  required  to  use  certain  ALIS  applications,  and  often  in 
different  ways  based  on  their  job  requirements.  Maintainers  at  two  of  the 
five  sites  said  that  the  classroom  training  reputation  has  become  so  bad 
that  many  new  maintainers  choose  to  skip  the  courses  and  proceed 
directly  to  on-the-job  training  when  they  begin  working  on  the  F-35. 


From  the  outset  of  the  original  ALIS  release  in  2010,  DOD  has  relied  on 
the  contractor  to  manage  all  ALIS  training.  ALIS  training  is  currently 
heavily  dependent  on  learning  on  the  job,  which  consists  of  users  learning 
how  to  operate  ALIS  from  colleagues  and  through  trial  and  error. 
According  to  DOD  officials,  this  is  not  an  uncommon  practice  for  new 
systems  within  the  department;  however,  those  officials  agree  that 
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training  typically  begins  with  a  common  classroom  curriculum,  and  is 
supplemented  with  on-the-job-training.  Almost  every  ALIS  user  in  our 
focus  groups  noted  that  they  do  not  learn  how  to  operate  any  ALIS 
applications  until  on-the-job-training  begins  on  the  flight  line.  Specifically, 
the  classroom  training  does  not  afford  an  opportunity  to  practice  on 
“sandbox”  or  “ghost”  systems  that  would  simulate  how  ALIS  is  used  on  a 
daily  basis  by  users.  Instead,  the  practice  comes  in  a  live  operational 
environment,  where  basic  ALIS  functions  and  practices  are  taught  by 
supervisors  or  other  users  within  the  respective  squadron.  According  to 
ALIS  users  at  all  five  sites,  this  practice  has  led  to  users  learning 
inconsistent  methods  and  shortcuts  for  maneuvering  through  the  different 
ALIS  applications.  In  most  cases,  these  practices  differ  between 
squadrons  and  services;  however,  there  were  also  cases  that  some 
maintainers  highlighted  where  users  were  learning  different  practices 
within  the  same  squadron.  ALIS  users  reported  that  they  have  created 
different  workarounds  to  overcome  ALIS  functionality  issues,  but  have 
also  cited  instances  of  not  using  ALIS  as  intended  because  the  system  is 
unwieldy  and  time-consuming.  ALIS  users  said  they  are  learning  to 
operate  the  system  in  different  ways  and  then  perpetuating  these 
methods,  creating  a  situation  where  ALIS  may  not  be  operated  in  the 
most  effective,  efficient,  or  up-to-date  way  across  all  F-35  sites. 
Furthermore,  they  are  learning  to  use  the  system  in  a  live  operational 
environment,  running  the  risk  of  making  errors  that  could  ultimately  affect 
aircraft  availability. 

Program  officials  acknowledged  the  training  shortfalls  identified  in  the 
focus-group  sessions.  In  response  to  these  shortfalls,  the  program  office 
has  taken  some  initial  steps  to  address  some  of  the  issues.  Specifically, 
the  program  office,  in  concert  with  the  contractor,  has  developed  Mobile 
T raining  T earns  to  offer  a  way  to  train  ALIS  users  outside  of  the  F-35 
Academic  Training  Center  and  at  their  specific  F-35  sites.  These  teams 
are  deployed  to  F-35  operational  and  testing  sites  to  help  keep  ALIS 
users  up-to-date  with  ALIS  software  version  releases,  which,  according  to 
ALIS  users  at  all  sites  we  visited,  had  been  a  significant  problem  with 
ALIS  training.  Specifically,  according  to  ALIS  users  at  all  these  sites,  ALIS 
trainers  and  administrators  are  rarely  up-to-date  on  the  latest  ALIS 
releases  and  functionality  changes;  therefore,  it  has  led  to  inconsistencies 
in  teachings  and  practices  at  the  squadron  level.  Mobile  Training  Teams 
offer  ALIS  version-specific  training  at  each  site  based  on  a  sequencing 
schedule  developed  by  the  program  office. 

In  addition,  the  program  office  has  rolled  out  an  ALIS  Training  Evaluation 
to  determine  the  current  state  of  ALIS  training  across  all  F-35  sites.  The 
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end  goal  of  this  evaluative  process  is  to  identify  underlying  training 
deficiencies,  and  to  develop  corrective  courses  of  action  to  mitigate  these 
deficiencies.  The  process  will  include  a  series  of  site  visits  to  include 
course  audits,  interviews,  curricula  inspections,  and  stakeholder  surveys 
as  applicable  for  root-cause  analysis.  As  of  January  2016,  a  team  within 
the  program  office  had  just  begun  this  process;  therefore,  they  could  not 
provide  any  details  beyond  the  effort’s  scope,  methodology,  and 
associated  time  frames. 

According  to  best  practices  of  information-technology  training,  effective 
training  of  users  is  essential  to  the  workforce  supporting  an  information- 
technology  system.42  The  practices  suggest  that  entities  develop 
Strategic  Learning  Plans,  or  Overarching  Training  Plans,  to  help  align 
training  programs  with  priorities.  Furthermore,  as  part  of  this  process,  the 
practices  state  that  it  is  important  that  the  training  design  and  delivery 
process  ensures  learning  occurs  during  the  training  and  also  ensures  that 
the  user  applies  the  training  on  the  job.  Additional  guidance  on  strategic 
training  in  the  federal  government  states  that  training  plans  can  aid  in  the 
performance  of  government  programs.43  Specifically,  a  training  plan  can 
present  a  business  case  for  proposed  training  and  development 
investments,  including  the  identified  problem  or  opportunity,  the  concept 
for  an  improved  situation  or  condition,  and  linkages  with  strategic 
objectives. 

According  to  DOD  officials,  ALIS  training  has  been  a  difficult  process  to 
manage  because  of  the  dynamic  nature  in  which  ALIS  has  been 
developed  and  upgraded  since  its  initial  release.  Because  new  versions 
of  ALIS  have  been  regularly  released  in  a  staggered  manner  across 
F-35  sites,  they  said  it  has  been  difficult  to  sufficiently  train  all  ALIS  users 
on  the  most  up-to-date  versions  and  teach  consistent  practices.  However, 
with  only  one  major  version  upgrade  remaining  prior  to  version  3.0  (the 
final  major  software  release),  issues  related  to  constantly  changing  the 
system  should  decrease.  A  standardized,  program-wide  training  plan 
could  remove  the  emphasis  from  on-the-job  training  and  provide  a 
comprehensive,  standardized  training  curriculum  across  the  program. 
Without  a  program-wide  training  plan  that  assures  that  consistent  learning 
occurs  in  the  classroom,  and  is  then  applied  by  users  on  the  job,  the 
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program  runs  the  risk  of  continuing  to  allow  users  to  learn  irregular  or 
incorrect  practices  through  a  training  culture  driven  by  on-the-job  training, 
which  could  impact  aircraft  availability  and  safety. 


Conclusions 


The  F-35  program  is  DOD’s  largest  and  most  costly  acquisition  program 
to  date.  According  to  senior  defense  officials,  not  only  does  the  F-35 
program  represent  the  future  of  tactical  fighter  aircraft  in  our  military,  but  it 
is  also  vital  to  the  security  of  our  nation  moving  forward.  With  the  Marine 
Corps  already  having  declared  the  F-35  both  deployable  and  combat- 
ready,  the  Air  Force  and  Navy  set  to  do  the  same  within  the  next  2  years, 
and  the  full-rate  production  of  the  aircraft  set  for  2019,  including  the  ramp 
up  of  sustainment  activities,  it  is  imperative  that  DOD  address  major  risks 
associated  with  its  central  logistics  system — ALIS.  The  program  office  has 
taken  steps  to  identify  risks  associated  with  ALIS,  including  all  of  those 
identified  by  participants  in  our  focus  groups — and  has  begun,  on  a  case- 
by-case  basis,  to  address  some  of  these  risks;  however,  without  a  plan  to 
prioritize  risks  and  address  them  in  a  systematic  and  holistic  manner, 

DOD  runs  the  risk  of  having  an  ALIS  that  is  not  fully  functional  as  it 
approaches  key  program  milestones.  Without  a  fully  functional  ALIS, 

DOD  could  face  operational  and  schedule  risks  and  potential  cost 
increases  to  a  program  that  is  already  the  most  expensive  in  DOD’s 
history. 

Although  DOD  has  estimated  the  costs  of  ALIS,  additional  information 
would  increase  the  accuracy  and  credibility  of  the  estimate.  While  ALIS  is 
projected  to  constitute  less  than  2  percent  of  the  $891 .1  billion  total 
sustainment  costs  of  the  program,  the  financial  impact  that  nonfunctional 
aspects  of  ALIS  may  have  on  the  overall  operations  and  sustainment  of 
the  aircraft  could  be  significant.  Until  DOD  does  more  analyses  to 
determine  the  impact  ALIS  has  (and  the  impact  a  nonfunctional  ALIS 
could  have)  on  overall  sustainment  costs,  DOD  will  not  know  how  much 
the  costs  of  ALIS  and  overall  sustainment  could  fluctuate.  Furthermore, 
while  the  program  office  has  been  incorporating  some  program  changes 
and  adjusting  its  cost-estimating  assumptions  for  ALIS,  it  is  important  for 
the  program  office  to  improve  the  reliability  of  its  estimate  by  using 
historical  data.  It  is  also  important  for  DOD  to  incorporate  program 
changes  that  will  likely  affect  ALIS  costs  in  future  estimates,  such  as 
decisions  to  enhance  ALIS  infrastructure  or  decrease  planned  numbers  of 
administrative  personnel  with  regional  support  centers. 

Finally,  although  DOD  has  recognized  that  ALIS  training  needs 
improvement  and  has  made  some  temporary  fixes  to  address  the  current 
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shortcomings,  DOD  has  yet  to  develop  a  program-wide  training  plan  that 
would  take  the  focus  off  an  almost  explicit  on-the-job-training  approach, 
and  provide  greater  consistency  among  ALIS  users.  Considering  the 
importance  ALIS  has  to  the  operations  and  sustainment  of  the  aircraft, 
and  that  DOD  plans  to  purchase  and  operate  nearly  500  more  F-35s  in 
the  next  5  years,  it  will  be  important  that  ALIS  users  be  on  the  same  page 
with  regard  to  operating  the  system.  Further  training  inconsistencies  could 
lead  to  impacts  on  aircraft  availability  and  safety. 


Recommendations  for 
Executive  Action 


We  recommend  that  the  Secretary  of  Defense  direct  the  F-35  Program 

Executive  Officer  to  take  the  following  four  actions: 

•  To  ensure  that  risks  associated  with  ALIS  are  addressed  expediently 
and  holistically,  develop  a  plan  that  would  prioritize  and  address  ALIS 
issues,  prior  to  the  start  of  full-rate  production  for  the  program. 

•  To  improve  the  reliability  of  its  cost  estimates,  conduct  uncertainty  and 
sensitivity  analyses  consistent  with  cost-estimating  best  practices 
identified  in  GAO’s  Cost  Estimating  and  Assessment  Guide. 

•  To  improve  the  reliability  of  its  cost  estimates,  ensure  that  future 
estimates  of  ALIS  costs  use  historical  data  as  available  and  reflect 
significant  program  changes  consistent  with  cost-estimating  best 
practices  identified  in  GAO’s  Cost  Estimating  and  Assessment  Guide. 

•  To  ensure  that  ALIS  training  issues  are  fully  addressed,  develop  a 
standardized,  program-wide  plan  for  ALIS  training  through  the  life 
cycle  of  the  program. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  concurred  with  two 
recommendations  and  partially  concurred  with  two  recommendations. 
DOD’s  comments  are  summarized  below  and  reprinted  in  appendix  III. 
DOD  also  provided  technical  comments,  which  we  have  incorporated  into 
our  report  where  appropriate. 

DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to  develop  a  plan  that  would 
prioritize  and  address  ALIS  issues,  prior  to  the  start  of  full-rate  production 
for  the  program.  DOD  stated  that  the  F-35  program  office  began 
developing  an  ALIS  Technical  Roadmap  in  early  2016.  The  department 
added  that  at  its  completion  later  in  2016,  this  roadmap  will  be  the 
foundation  of  a  plan  to  identify,  document,  and  prioritize  ALIS  risks; 
address  them  holistically;  and  inform  budget  priorities,  as  the  program 
transitions  from  development  into  sustainment  and  follow-on 
modernization.  Additionally,  to  mitigate  the  risk  of  single-point  failures  in 
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the  infrastructure,  the  program  office  contracted  to  acquire  backup  ALIS 
hardware  in  2015;  the  backup  hardware  will  be  operational  by  early  2017. 
We  state  in  our  report  that  the  department  was  aware  of  the  risks  to  ALIS 
and  believe  that  if  DOD  develops  a  plan  to  identify,  document,  and 
prioritize  ALIS  risks,  address  them  holistically,  and  inform  budget  priorities 
prior  to  full-rate  production,  this  action  should  address  our 
recommendation.  We  further  state  in  our  report  that  DOD  was  in  the  early 
stages  of  acquiring  backup  ALIS  hardware  to  mitigate  the  risk  of  single¬ 
point  failures  in  the  infrastructure,  but  had  not  yet  allotted  funding.  We 
believe  this  backup  system  will  be  critical  as  the  program  approaches  full- 
rate  production. 

DOD  partially  concurred  with  the  recommendation  that  the  Secretary  of 
Defense  direct  the  F-35  Program  Executive  Officer  to  conduct  uncertainty 
and  sensitivity  analyses  consistent  with  cost-estimating  best  practices 
identified  in  GAO’s  Cost  Estimating  and  Assessment  Guide.  DOD  stated 
that  the  department  considers  the  sensitivity  analyses  that  the  F-35 
program  office  performs  to  be  a  form  of  uncertainty  analysis,  as  described 
in  DOD’s  Cost  Assessment  and  Program  Evaluation  Operating  &  Support 
Cost  Estimating  Guide ;  however,  the  DOD  cost  estimating  guidance  does 
not  require  DOD  to  conduct  a  sensitivity  or  uncertainty  analysis  on  ALIS 
since  DOD  does  not  consider  ALIS  a  major  cost  driver  of  the  F-35 
program.  As  our  report  states,  according  to  GAO’s  Cost  Estimating  and 
Assessment  Guide,  cost  estimates  should  include  uncertainty  analyses  to 
determine  the  level  of  uncertainty  associated  with  the  estimate  in  order  to 
be  credible.  A  quantitative  uncertainty  analysis  can  provide  a  broad 
overall  assessment  of  the  risk  in  the  cost  estimate.  A  sensitivity  analysis 
can  examine  how  changes  to  individual  assumptions  and  inputs  affect  the 
estimate  as  a  whole.  Although  the  F-35  program  office  may  conduct 
analyses  consistent  with  DOD  cost  estimating  guidance,  it  has  not 
conducted  an  uncertainty  or  sensitivity  analysis  specifically  for  ALIS. 
Although  ALIS  is  projected  to  constitute  less  than  2  percent  of  the  $891 .1 
billion  total  sustainment  costs  of  the  program,  the  financial  impact  that 
nonfunctional  aspects  of  ALIS  may  have  on  the  overall  operations  and 
sustainment  of  the  aircraft  could  be  significant.  For  example,  a  2013 
DOD-commissioned  plan  found  that  any  functionality  problems  or 
schedule  slippage  with  ALIS  will  have  a  significant  impact  on  costs — with 
downstream  additional  costs  due  to  performance  and  schedule  delays 
potentially  reaching  up  to  $20-100  billion.  We  continue  to  believe  that 
without  completing  uncertainty  and  sensitivity  analyses  to  determine  the 
effect  ALIS  has  (and  the  impact  a  nonfunctional  ALIS  could  have)  on 
overall  sustainment  costs,  DOD  will  not  know  how  much  the  costs  of  ALIS 
and  overall  sustainment  could  fluctuate. 
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DOD  partially  concurred  with  the  recommendation  that  the  Secretary  of 
Defense  direct  the  F-35  Program  Executive  Officer  to  ensure  that  future 
estimates  of  ALIS  costs  use  historical  data  as  available  and  reflect 
significant  program  changes  consistent  with  cost-estimating  best 
practices  identified  in  GAO’s  Cost  Estimating  and  Assessment  Guide. 
DOD  stated  that  the  department  will  ensure  that  the  future  F-35  program 
ALIS  cost  estimates  continue  to  use  the  latest  available  historical  cost 
data  as  appropriate  and  reflect  the  latest  approved  technical  baseline 
when  the  program  office  incorporates  these  into  the  program  of  record, 
according  to  DOD’s  Cost  Assessment  and  Program  Evaluation  Operating 
and  Support  Cost  Estimating  Guide;  however,  we  found  that  DOD  was 
not  using  all  available  historical  data.  As  our  report  states,  according  to 
GAO’s  Cost  Estimating  and  Assessment  Guide,  a  cost  estimate  should 
be  based  on  historical  data — both  actual  costs  of  the  program  and  those 
of  comparable  programs — which  can  be  used  to  challenge  optimistic 
assumptions  and  bring  more  realism  to  a  cost  estimate.  While  we  found 
that  DOD  substantially  met  some  best  practices  for  an  accurate  cost 
estimate  by  properly  adjusting  for  inflation  and  not  including  mathematical 
errors,  the  estimate  uses  contractor-provided  data  for  material  costs 
instead  of  actual  ALIS  costs  or  historical  cost  data  from  analogous 
programs  that  would  make  the  estimate  more  accurate.  We  continue  to 
believe  that  it  is  important  for  the  program  to  improve  the  reliability  of  its 
ALIS  estimate  by  using  historical  data  to  the  greatest  extent  possible.  It  is 
also  important  for  DOD  to  incorporate  program  changes  that  will  likely 
affect  ALIS  costs  in  future  estimates,  such  as  decisions  to  enhance  ALIS 
infrastructure  or  decrease  planned  numbers  of  administrative  personnel 
for  regional  support  centers. 

DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to  develop  a  standardized, 
program-wide  plan  for  ALIS  training  through  the  life  cycle  of  the  program. 
DOD  stated  that,  to  address  immediate  issues,  the  F-35  program  office 
deployed  Mobile  Training  Teams  to  assist  ALIS  users  at  their  home  base 
locations,  and  to  address  longer-term  issues,  the  program  office  began  a 
comprehensive  evaluation  of  ALIS  training  in  2015.  According  to  DOD, 
the  completion  of  this  evaluation  in  2016  will  inform  development  of  a  plan 
to  address  long-term  ALIS  training  issues.  We  agree  and  state  in  our 
report  that  the  F-35  program  has  taken  some  positive  steps  to  address 
short-term  training  shortfalls  by  deploying  Mobile  Training  Teams  as  a 
way  to  train  ALIS  users  outside  of  the  F-35  Academic  T raining  Center. 

We  also  report  that  the  program  has  recently  begun  a  comprehensive 
evaluation  of  ALIS  training  to  determine  the  current  state  of  ALIS  training 
across  all  F-35  sites.  If  the  F-35  program  leverages  this  comprehensive 
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evaluation,  when  it  is  completed,  to  develop  a  program-wide  plan  for  ALIS 
training  through  the  life  cycle  of  the  program,  this  action  should  address 
our  recommendation. 


We  are  sending  copies  of  this  report  to  appropriate  congressional 
committees;  the  Secretary  of  Defense;  the  Secretaries  of  the  Air  Force, 
Army,  and  Navy;  and  the  Commandant  of  the  Marine  Corps.  In  addition, 
the  report  is  available  at  no  charge  on  the  GAO  website  at 

http://www.gao.gov. 

If  you  or  your  staff  has  any  questions  about  this  report,  please  contact  me 
at  (202)  512-5431  or  russellc@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  Staff  members  making  key  contributions  to  this  report  are 
listed  in  appendix  IV. 


Cary  Russell,  Director 

Defense  Capabilities  and  Management 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 

The  Honorable  Michael  R.  Turner 
Chairman 

The  Honorable  Loretta  L.  Sanchez 
Ranking  Member 

Subcommittee  on  Tactical  Air  and  Land  Forces 
Committee  on  Armed  Services 
House  of  Representatives 
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To  determine  the  extent  to  which  the  Department  of  Defense  (DOD)  has  a 
plan  to  ensure  the  Autonomic  Logistics  Information  System  (ALIS)  is  fully 
functional  as  the  key  F-35  program  milestones  approach,  we  reviewed 
documentation  of  program  plans  with  relevant  sustainment  elements 
including  the  F-35  Global  Sustainment  Plan,  the  Weapon  System 
Planning  Document,  the  F-35  Autonomic  Logistics  Global  Sustainment 
Concept  of  Operations,  and  the  F-35  Operational  Requirements 
Document.  We  also  selected  and  conducted  site  visits  at  a 
nongeneralizable  sample  of  five  F-35  operational  and  training  sites: 

•  Eglin  Air  Force  Base,  Florida 

•  Luke  Air  Force  Base,  Arizona 

•  Edwards  Air  Force  Base,  California 

•  Nellis  Air  Force  Base,  Nevada 

•  Marine  Corps  Air  Base  Yuma,  Arizona 

We  selected  these  sites  in  consultation  with  service  officials  to  ensure  we 
obtained  perspectives  across  all  three  services  and  at  both  operational 
and  testing  sites.1  During  these  visits,  we  convened  17  non-generalizable 
focus-group  sessions  with  a  range  of  ALIS  users  from  all  three  services  to 
obtain  information  on  the  operability  and  deployability  of  ALIS,  and  how 
any  ALIS  issues  may  pose  risks  for  F-35  operations  and  sustainment. 
Specifically,  we  convened  groups  of  maintainers,  pilots,  system 
administrators,  and  trainers.  We  also  held  focus  groups  with  contractor 
personnel  responsible  for  training  and  administering  ALIS  at  these  sites. 
There  were  approximately  a  total  of  120  participants  in  these  focus 
groups.  Table  3  includes  a  breakdown  of  the  focus  groups  we  held  at  the 
various  locations. 


^hese  locations  included  Air  Force  and  Marine  Corps  bases.  In  addition,  although  we  did 
not  visit  a  Navy  installation,  we  spoke  with  Navy  personnel  working  in  the  F-35  program  at 
one  of  these  locations. 
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Table  3:  Focus  Groups  Convened  at  F-35  Sites 


Location 

Focus  groups  convened 

Eglin  Air  Force  Base 

•  Pilots,  maintainers,  trainers,  administrators 

Luke  Air  Force  Base 

•  Pilots,  maintainers,  trainers,  administrators 

Marine  Corps  Air  Base  Yuma 

•  Pilots,  maintainers,  administrators 

Edwards  Air  Force  Base 

•  Pilots,  maintainers,  administrators 

Nellis  Air  Force  Base 

•  Pilots,  maintainers,  administrators 

Source:  GAO  I  GAO-16-439 


We  worked  with  our  methodologist  to  develop  a  focus  group  script  that 
included  questions  across  four  main  categories:  Training,  ALIS  Positives, 
ALIS  Negatives,  and  Risks,  that  was  used  at  all  five  site  visits.  For 
consistency,  our  methodologist  facilitated  all  focus  group  sessions.  To 
analyze  the  focus-group  responses,  we  conducted  content  analyses, 
developing  categories  and  sub-categories  and  coding  comments  from 
each  focus  group  to  these  categories.  After  each  comment  had  been 
coded  by  an  analyst,  another  analyst  independently  reviewed  each  code 
and  either  agreed  or  disagreed  with  the  coding  decision.  Where  there  was 
disagreement  in  the  coding  decision,  the  two  analysts  discussed  it  and 
came  to  a  resolution.  Based  on  the  content  analysis,  we  described  the 
overarching  ALIS  benefits  and  risks  obtained  from  ALIS  users  about  the 
system  and  any  risks  it  poses  to  the  F-35  program  as  it  approaches  key 
milestones.  After  obtaining  information  on  DOD’s  current  approach  to 
addressing  the  functionality  issues  that  ALIS  users  identified,  we 
evaluated  information  we  obtained  for  consistency  with  best  practices 
from  GAO’s  Schedule  Assessment  Guide  and  DOD’s  System 
Engineering  Guide  for  System  of  Systems  that  provide  guidance  and  best 
practices  on  how,  prior  to  meeting  key  milestones,  a  plan  to  address 
specific  risks  that  may  be  associated  with  major  weapon  acquisitions 
should  be  developed.2  We  also  interviewed  key  DOD  and  contractor 
officials  to  collect  information  about  building  and  testing  ALIS,  the 
capabilities  of  ALIS,  metrics  collected  on  ALIS’s  development  and 
performance,  software  upgrades  to  ALIS,  and  how  the 
F-35  program  office  is  addressing  ALIS  functionality  issues. 


2GAO,  Schedule  Assessment  Guide,  GAO-12-120G  (Washington,  D.C.:  May  2012)  and 
Department  of  Defense,  Systems  Engineering  Guide  for  System  of  Systems  (Washington, 
D.C.:  August  2008). 
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To  determine  the  extent  to  which  DOD  has  credibly  and  accurately 
estimated  ALIS  costs,  we  evaluated  the  reliability  of  DOD’s  estimate  of 
ALIS  costs  contained  in  the  F-35  program  office’s  2014  estimate  of  F-35 
operating  and  support  (O&S)  costs,  the  most  up-to-date  estimate 
completed  at  the  time  of  our  review.  The  program  office  completed  the 
2014  cost  estimate  in  the  spring  of  2015  and  plans  to  release  its  2015 
updated  estimate  in  spring  of  2016.  The  Office  of  the  Director  for  Cost 
Assessment  and  Program  Evaluation  (CAPE)  also  performed  an  F-35 
O&S  cost  estimate  in  2013,  but  has  not  updated  it.  Both  CAPE  and  Joint 
Program  Office  officials  told  us  they  used  the  same  cost  inputs, 
methodology,  and  ground  rules  and  assumptions  when  estimating  ALIS 
costs  in  their  respective  estimates.  We  used  characteristics  contained  in 
GAO’s  Cost  Estimating  and  Assessment  Guide  to  assess  the  reliability  of 
DOD’s  estimate  of  ALIS  costs.3  According  to  the  guide,  there  are  four 
general  characteristics  of  sound  cost  estimating:  being  well-documented, 
comprehensive,  credible,  and  accurate.  For  the  purposes  of  this  review, 
we  conducted  a  limited  assessment  and  used  two  of  the  four  general 
characteristics  of  sound  cost  estimating  included  in  this  guide:  being 
credible  and  accurate.  We  chose  these  characteristics  because  ALIS 
costs  represent  only  one  element  of  the  total  F-35  cost  estimate — less 
than  2  percent  of  projected  F-35  sustainment  costs — and  therefore  we 
determined  that  it  would  not  be  appropriate  to  assess  whether  the 
estimate  was  comprehensive  or  well-documented.  To  determine  whether 
the  credible  and  accurate  characteristics  were  met,  we  reviewed 
documentation  used  to  generate  the  program  office’s  estimate,  including 
data  sources,  assumptions,  and  cost  models,  and  we  interviewed  cost¬ 
estimating  officials  from  the  program  office  and  CAPE.  We  also 
interviewed  other  officials  from  the  program  office  and  service 
headquarters  to  discuss  the  cost  effects  of  ALIS  schedule  delays  and 
development  issues.  Results  of  our  assessment  of  the  estimate’s 
credibility  and  accuracy,  along  with  descriptions  of  these  characteristics 
and  their  associated  best  practices,  are  detailed  in  appendix  II  of  this 
report.  We  found  the  data  to  not  be  fully  reliable,  which  we  discuss  in 
further  detail  in  the  report  and  in  appendix  II. 


3GAO,  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs ,  GAO-09-3SP  (Washington,  D.C.:  March  2009).  The 
methodology  outlined  in  the  Cost  Estimating  and  Assessment  Guide  is  a  compilation  of 
best  practices  that  federal  cost-estimating  organizations  and  industry  use  to  develop  and 
maintain  reliable  cost  estimates  throughout  the  life  of  an  acquisition  program. 
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Finally,  to  determine  the  extent  to  which  DOD  has  developed  a  plan  to 
manage  ALIS  training  for  users,  we  reviewed  key  documentation  related 
to  ALIS  and  F-35  training,  and  used  information  from  our  focus-group 
sessions  across  various  types  of  ALIS  users  from  all  three  services  to 
obtain  information  on  the  current  state  of  ALIS  training.  We  also 
interviewed  key  DOD  and  contractor  officials.  We  evaluated  all  of  the 
information  we  received  using  GAO-developed  and  industry  best 
practices  for  information-technology  training,  and  DOD’s  Policies  and 
Procedures  for  Acquisition  of  Information  Technology.4 

To  address  all  of  our  objectives,  we  collected  and  analyzed  information 
and  interviewed  officials  from  the  following  Department  of  Defense  (DOD) 
offices: 

•  Office  of  the  Linder  Secretary  of  Defense  (Acquisitions,  Technology 
and  Logistics) 

•  Office  of  the  Director  for  Cost  Assessment  and  Program  Evaluation 
(CAPE) 

•  Office  of  the  Director  for  Operational  Test  and  Evaluation  (DOT&E) 

•  Department  of  the  Air  Force 

•  Department  of  the  Navy 

•  Headquarters  Marine  Corps 

•  F-35  Joint  Program  Office 

We  also  collected  and  analyzed  information  and  interviewed  officials  from 
Lockheed  Martin  in  Fort  Worth,  Texas,  and  Orlando,  Florida. 

We  conducted  this  performance  audit  from  April  2015  to  April  2016  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  and  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


4GAO,  Information  Technology  Training:  Practices  of  Leading  Private-Sector  Companies, 
GAO-03-390  (Washington,  D.C.:  Jan.  31, 2003);  and  Human  Capital:  A  Guide  for 
Assessing  Strategic  Training  and  Development  Efforts  in  the  Federal  Government, 
GAO-04-546G  (Washington,  D.C.:  March  2004). 
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We  completed  an  assessment  of  the  ALIS  costs  in  the  F-35  program 
office  overall  F-35  O&S  cost  estimate  on  the  basis  of  two 
characteristics — being  credible  and  accurate — and  their  associated  best 
practices  derived  from  the  GAO  Cost  Estimating  and  Assessment  Guide. 
After  reviewing  documentation  that  the  program  office  submitted  for  its 
2014  F-35  ALIS  O&S  cost  estimate,  conducting  interviews  with  program 
office  cost-estimating  officials,  and  reviewing  relevant  sources,  we 
determined  that  these  cost  estimates  are  not  fully  reliable.  While  we  found 
that  the  program  office  estimate  for  ALIS  is  partially  accurate,  the 
estimate  is  minimally  credible.  These  evaluations  are  shown  in  table  4. 

We  determined  the  overall  assessment  rating  by  assigning  each 
individual  best  practice  rating  a  number:  Not  Met  =  1,  Minimally  Met  =  2, 
Partially  Met  =  3,  Substantially  Met  =  4,  and  Met  =  5.  Then,  we  took  the 
average  of  the  individual  best  practice  assessment  ratings  to  determine 
the  overall  rating  for  each  of  the  two  characteristics.  The  resulting 
average  becomes  the  Overall  Assessment  as  follows:  Not  Met  =  1.0  to 
1 .4,  Minimally  Met  =  1 .5  to  2.4,  Partially  Met  =  2.5  to  3.4,  Substantially 
Met  =  3.5  to  4.4,  and  Met  =  4.5  to  5.0.  A  cost  estimate  is  considered 
reliable  if  the  overall  assessment  ratings  for  each  of  the  two 
characteristics  are  substantially  or  fully  met.  If  any  of  the  characteristics 
are  not  met,  minimally  met,  or  partially  met,  then  the  cost  estimate  does 
not  fully  reflect  the  characteristics  of  a  high-quality  estimate  and  cannot 
be  considered  reliable. 


Table  4:  Summary  Assessment  of  F-35  Program  Office  Estimate  of  Autonomic  Logistics  Information  System  (ALIS)  Costs 


Characteristic 

Overall 

assessment 

Best  practice3 

Individual 

assessment 

Credible 

Minimally  met 

The  cost  estimate  includes  a  sensitivity  analysis  that  identifies  a  range  of 
possible  costs  based  on  varying  major  assumptions,  parameters,  and 
data  inputs. 

Not  met 

A  risk  and  uncertainty  analysis  was  conducted  that  quantified  the 
imperfectly  understood  risks  and  identified  the  effects  of  changing  key 
cost  driver  assumptions  and  factors. 

Not  met 

Major  cost  elements  were  cross-checked  to  see  whether  results  were 
similar. 

Not  metb 

An  independent  cost  estimate  was  conducted  by  a  group  outside  the 
acquiring  organization  to  determine  whether  other  estimating  methods 
produce  similar  results. 

Partially  metc 

Accurate 

Partially  met 

The  cost  estimate  results  are  unbiased,  not  overly  conservative  or 
optimistic  and  based  on  an  assessment  of  most  likely  costs. 

Not  met 

The  estimate  has  been  adjusted  properly  for  inflation. 

Substantially  met 

The  estimate  contains  few,  if  any,  minor  mistakes 

Substantially  met 
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Office's  2014  Autonomic  Logistics  Information 

System  (ALIS)  Operating  and  Support  (O&S) 

Cost  Estimate 

Overall 

Characteristic  assessment 

Best  practice3 

Individual 

assessment 

The  cost  estimate  is  regularly  updated  to  reflect  significant  changes  in 
the  program  so  that  it  is  always  reflecting  current  status. 

Substantially  met 

Variances  between  planned  and  actual  costs  are  documented, 
explained,  and  reviewed. 

n/ad 

The  estimate  is  based  on  a  historical  record  of  cost  estimating  and  actual 
experiences  from  other  comparable  programs. 

Minimally  met 

The  estimating  technique  for  each  cost  element  was  used  appropriately. 

Minimally  met® 

Source:  GAO  analysis  of  F-35  program  office  data.  |  GAO-16-439 


Note:  n/a  =  not  applicable. 

aBest  practices  are  derived  from  GAO,  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for 
Developing  and  Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

bThe  F-35  program  office  told  us  it  uses  cross-checks,  but  we  could  not  confirm  this  through  analysis 
of  the  documentation  it  provided. 

cThe  Department  of  Defense’s  (DOD)  Office  of  the  Director  for  Cost  Assessment  and  Program 
Evaluation  (CAPE)  conducted  an  independent  cost  estimate  of  the  F-35  program  in  2013  that 
includes  ALIS  costs,  but  this  estimate  has  not  been  updated  since  then. 

dThe  F-35  program  office  did  not  provide  actual  cost  data  as  a  basis  for  the  estimate  and  we  were, 
therefore,  unable  to  perform  variance  analysis. 

eThe  F-35  program  office  did  not  provide  data  that  would  allow  us  to  determine  the  appropriateness  of 
the  methodology  used  to  estimate  ALIS  costs. 
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ASSISTANT  SECRETARY  OF  DEFENSE 

3500  DEFENSE  PENTAGON 
WASHINGTON,  DC  2030 1  -3500 


MAfi  2  5  2016 


Mr.  Cary  Russell 

Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr,  Russell: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  Government  Accountability 
Office  (GAO)  Draft  Report,  GAO- 16-439,  “F-35  SUSTAINMENT:  DOD  Needs  a  Plan  to 
Address  Risks  Related  to  Its  Central  Logistics  System”  dated  March  14,  2016  (GAO  Code 
352004).  Detailed  comments  on  the  report  recommendations  are  enclosed. 


Sincerely, 


David  J.  Berteau 


Enclosure: 
As  stated 


LOGISTICS  AND 
MATERIEL  READINESS 
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GAO  Draft  Report  Dated  March  14,  2016 
GAO-16-439  (GAO  CODE  352004) 

“F-35  SUSTAINMENT:  DOD  NEEDS  A  PLAN  TO  ADDRESS  RISKS  RELATED  TO  ITS 
CENTRAL  LOGISTICS  SYSTEM” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  To  ensure  that  risks  associated  with  ALIS  are  addressed  expediently 
and  holistically,  the  GAO  recommends  that  the  Secretary  of  Defense  direct  the  F-35  Program 
Executive  Officer  develop  a  plan  that  would  prioritize  and  address  ALIS  issues,  prior  to  the  start 
of  full-rate  production  for  the  program. 

DoD  RESPONSE:  Concur 

The  Department  concurs  with  the  recommendation  to  address  ALIS  risks.  The  F-35  Program 
Executive  Officer  (PEO)  began  developing  an  ALIS  Technical  Roadmap  in  early  2016.  At  its 
completion  later  in  2016,  this  effort  will  be  the  foundation  of  a  plan  to  identify,  document  and 
prioritize  ALIS  risks,  address  them  holistically  and  inform  budget  priorities,  as  the  program 
transitions  from  development,  into  sustainment  and  follow-on  modernization.  Additionally,  to 
mitigate  the  risk  of  single  point  failures  in  the  infrastructure,  the  PEO  contracted  to  acquire 
backup  ALIS  hardware  in  2015;  the  backup  hardware  will  be  operational  by  early  2017. 

RECOMMENDATION  2:  To  improve  the  reliability  of  its  cost  estimates,  the  GAO 
recommends  that  the  Secretary  of  Defense  direct  the  F-35  Program  Executive  Officer  conduct 
uncertainty  and  sensitivity  analyses  consistent  with  cost  estimating  best  practices  identified  in  the 
GAO  Cost  Estimating  and  Assessment  Guide. 

DoD  RESPONSE:  Partially  Concur 

The  Department  partially  concurs  with  the  recommendation  to  improve  the  reliability  of  its  cost 
estimates  through  uncertainty  and  sensitivity  analysis.  The  Department  considers  the  sensitivity 
analyses  that  the  F-35  PEO  regularly  performs  to  be  a  form  of  uncertainty  analysis,  as  described 
in  the  2014  Office  of  the  Secretary  of  Defense  (OSD),  Cost  Assessment  and  Program  Evaluation 
(CAPE)  Operating  &  Support  (O&S)  Cost-Estimating  Guide,  Section  5.3.5.  The  Department 
will  ensure  that  PEO  continues  to  perform  sensitivity  analyses  according  to  DoD  guidance. 

RECOMMENDATION  3:  To  improve  the  reliability  of  its  cost  estimates,  the  GAO 
recommends  that  the  Secretary  of  Defense  direct  the  F-35  Program  Executive  Officer  ensure  that 
future  estimates  of  ALIS  costs  use  historical  data  as  available  and  reflect  significant  program 
changes  consistent  with  cost  estimating  best  practices  identified  in  the  GAO  Cost  Estimating  and 

Assessment  Guide. 

DoD  RESPONSE:  Partially  Concur 

The  Department  partially  concurs  with  the  recommendation  to  improve  the  reliability  of  its  cost 
estimates  through  using  historical  data  and  incorporating  program  changes.  The  Department  will 
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ensure  that  future  F-35  PEO  ALIS  cost  estimates  continue  to  use  the  latest  available  historical 
cost  data  as  appropriate  and  reflect  the  latest  approved  technical  baseline  when  the  PEO 
incorporates  these  into  the  program  of  record,  according  to  DoD  guidance  in  the  2014  OSD 
CAPE  O&S  Cost-Estimating  Guide. 

RECOMMENDATION  4:  To  ensure  that  ALIS  training  issues  are  fully  addressed,  the  GAO 
recommends  that  the  Secretary  of  Defense  direct  the  F-35  Program  Executive  Officer  develop  a 
standardized,  program-wide  plan  for  ALIS  training  through  the  life  cycle  of  the  program. 

DoD  RESPONSE:  Concur 

The  Department  concurs  with  the  recommendation  to  address  ALIS  training  issues.  To  address 
immediate  issues,  the  F-35  PEO  deployed  Mobile  Training  Teams  to  assist  ALIS  users  at  their 
home  base  locations.  To  address  longer-tenn  issues,  the  PEO  began  a  comprehensive  evaluation 
of  ALIS  training  in  2015.  At  its  completion  in  2016,  the  evaluation  will  infonn  development  of 
a  plan  to  address  long-term  ALIS  training  issues. 
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